Based on more than ten years of experience in the construction of electricity information collection systems and the challenges posed by the new situation of market-oriented power reform, Neusoft Company has built a comprehensive energy information collection platform to unify the collection and processing of electricity information from the purchase side, supply side, and sales side. It supports the direct collection of large and small specialized transformer terminals, concentrators, and communication function energy meters, and comprehensively realizes the collection, processing, storage, and application of electricity information data for various users, including small hydropower, small thermal power, photovoltaic power generation, and substation metering points. At the same time, the comprehensive energy information collection platform can achieve centralized collection, processing, storage, and release of water, electricity, heat, and gas.

The application architecture of Neusoft Comprehensive Energy Information Collection Platform can be divided into three levels from bottom to top: "Support Platform", "Collection Platform", and "Business Application", as shown in the following figure:

Supporting PlatformAs a basic supporting application, it includes Neusoft's unified integrated development platform, interface interaction platform, workflow support function, personnel and permission system management, alarm engine, public SMS platform, email system, etc.

Collection platformDivided into front-end communication functional domain and data processing functional domain. The front-end communication system consists of communication servers, management servers, load balancing devices, etc. Complete terminal access, protocol message reception, parsing, combining, message packaging, data storage, load balancing, and other functions. The data processing completes functions such as configuring the collection protocol, configuring the collection content, managing the collection strategy, issuing instructions, determining message integrity, and calculating initial data.

Business application layerAbove the platform support layer and collection platform is the comprehensive energy information collection platform, which provides various application functions for power enterprises. In addition to basic functions such as file management, terminal management, collection task management, data processing, data query, etc., it also includes necessary functions for stable system operation, such as operation monitoring, operation and maintenance management, etc. At the same time, the comprehensive energy information collection platform provides higher-level business management and big data analysis applications, such as cost control management, line loss management, electricity analysis, load analysis, load forecasting, abnormal analysis of metering equipment, etc.

Neusoft Company has summarized years of experience in the construction of information technology such as power information collection and power marketing, adopted a service-oriented SOA design concept, and optimized the architecture and upgraded the existing Neusoft Company power user electricity information collection system from the perspective of technical and business specialization design. Technically, it can meet the collection needs of small power sales companies (power supply enterprises), and can also smoothly expand to the data collection, processing, storage, and application needs of prefecture level and even provincial-level power grids; In terms of business, loose coupling deployment between modules can be achieved, enabling software to be split and combined according to application requirements, and enabling data exchange based on the IEC CIM model that meets international standards.

When targeting small and medium-sized power supply enterprises with terminal scale below 100000 and user scale below 2 million, the technical architecture adopted by the comprehensive energy information collection platform (all based on open-source software platform components) is as follows:

Real time data bus is used to decouple the acquisition components, which can be flexibly deployed according to requirements.

When the user base reaches millions or more, components such as real-time data bus are replaced with distributed computing and storage to achieve higher business performance. The technical architecture is shown in the following figure:

Technical Characteristics

High performance data collection and processing capabilities

High frequency data requirements require high-performance data collection capabilities, convenient instruction issuance, high-performance protocol parsing, and high-speed data storage and processing

• Introducing parallel processing and distributed caching technology to enhance the processing capability of collected data
• Introducing a distributed message queue to enhance the message exchange capability between the main station and the front-end processor
• Introducing distributed data storage to solve database I/O bottlenecks

Differentiated data management capabilities

High density transmission and publication of various collected data, therefore, how to manage collected data and efficiently publish data is a new application requirement for the collection system

• Introducing a distributed HBase database to solve the big data storage bottleneck of traditional relational databases
• The global large-scale HBase cluster (Facebook) undergoes 1.5 million operations per second, with a monthly increase of 250TB
• Based on distributed database technology (redundant data storage, etc.), automatic sharing of data reading load can be achieved
• Query a low voltage household, freeze data for a month, respond in milliseconds
• Real time data is written to the distributed cache on the same day, providing high-frequency data access to the trading center. The reading speed reaches 110000 times/second, and the writing speed reaches 81000 times/second
• Based on distributed message queues, a single server can provide read and write capabilities of 100Mbps data per second for thousands of clients simultaneously

Using Hadoop HDFS to achieve distributed storage of data

• It is a highly fault-tolerant distributed file system designed to run on low-cost hardware.
• HDFS provides high-throughput application data access capabilities, suitable for applications with large datasets
• HBase is a highly reliable, high-performance, column oriented, and highly scalable distributed column storage system built on HDFS; HBase is a typical key/value system;
• Used for storing massive structured data by continuously increasing the availability of low-cost commercial servers to enhance computing and storage capabilities.

High performance data analysis and mining capabilities

Complete data calculation, analysis, and mining in a short period of time

• Implementing parallel computing based on Map Reduce to improve the performance of data analysis and processing
• Distributed data storage reduces single node I/O bottlenecks and improves read efficiency
• Implementing complex data mining such as clustering analysis based on Spark memory computing
• Taking online monitoring and intelligent diagnostic analysis as an example, based on distributed computing and distributed storage technology, all 29 analysis topics were calculated within 1 hour, increasing efficiency by 10 times
• Can improve computing performance through horizontal scaling

Efficient data publishing and exchange capabilities

• Transitioning from 'data extraction' to 'data publishing'
• Based on distributed data storage, provide high concurrency and high-frequency data access
• Provide a web subscription method to achieve true real-time synchronous monitoring with the collection system
• Based on distributed message queues, ensure the timeliness and timing of power outage and restoration instructions
• Encryption of key information exchange (control type) between external units