Logs
We could store all the data generated by the HCS in a database but, this would generate a huge amount of data and is a bit over the top for the purpose and usage we intended for this data. To keep things nice and simple, our Home Control System (HCS) logs debugging, sensor and event data to text files. We generate one file per day and store each one in a folder for the month and then this sits in a folder for each year. E.g.
/Logs/2012/01/22/2012_01_22_HCS.txt
Typically, each daily log file is can be between 30Kb and 80Kb. Writing so much data and so frequently to hard disk is not good from a longevity and power consumption perspective, so the data is stored on a USB flash drive and backed up to hard disk once a day.
Log Format
Log data the form: 2010/09/23 18:45:56 E Description message text. After the time and data a single character is used to identify the type of log entry. The characters currently used are:
- !
- Process startup or shutdown
- .
- Initialisation and initial values on process startup
- E
- Error, e.g. Failed to control a device
- W
- Warning, e.g. home network speed is low
- C
- User command received, e.g. alarm on/off
- D
- Debug message
- O
- Our HCS has changed an output status to control something.
- -
- Device value change, e.g. room temperature, wind direction, wind speed, etc.
- =
- Environmental change, e.g. sunset, sunrise, dusk, dawn, mains power failure
Start & End Of Day
The logs are processed to support graphs and other data displays (more to come on this soon). To simplify the plotting of daily data, the HCS automatically dumps values into the log file at the start and end each day. This means that the data in each log file nicely spans the complete day when plotted out. This is thus also true for logs files aggregated to provide a monthly view too.
Temperature Data
The Dallas DS1920 temperature sensors on our 1-Wire are set to 0.5°C resolution. This approach was used as we continually poll the sensors, to detect changes in temperature. Initially, we logged every change detected but this generated a lot of log data. The main culprit behind all this data was hysteresis, because the reported temperature was seen to oscillate between two values as it approached a transition point.
The original objective behind this continual polling was to rapidly detect changes in temperature because these sensors are also used for fire detection. Rapid detection of fast rising temperatures will provide a very early warning of a potential fire.
The key issue to be resolved though was the amount of logged data. It was quickly realised that an accuracy of 0.5°C is excessive in terms of logging room temperatures though and if we ignored any values that were not integers, we could resolved the hysteresis problem and lower the amount of data logged significantly. This was the approach adopted through some very simple Java code.
Debugging
The logs are essential for debugging and experimenting. Changes made to the Home Control System (HCS) code may have impact over long periods of time, such as room temperature controllers. The logging of the data enables us to form a view of how well the changes work.
Usage & Analysis
We log a whole load of data in each day and it provides real insight into how things are working in our home and how and when things get used. There maybe more data than we know what to do with but, we are continually writing tools to process and analyse it and also have a project to visualise the log data.
