<<< Adding a Ecowitt Data Source to WeeWX
Add Ecowitt Data to WeeWX
The final step in the process is the add the new data source into the WeeWX database, but before we can do that we need to understand the contents of the files being uploaded
If we look at a data file we can see that the temperatures and humidity are contained in ab Array, I have obfuscated PASSKEY for obvioius reasons
Array
(
[PASSKEY] => ################################
[stationtype] => GW1000B_V1.6.3
[dateutc] => 2021-01-16 12:30:13
[tempinf] => 66.4
[humidityin] => 42
[baromrelin] => 28.981
[baromabsin] => 28.981
[tempf] => 40.3
[humidity] => 79
[wh26batt] => 0
[freq] => 915M
[model] => GW1000_Pro
)
Create Python Script to Parse the File
The data is well structured and so I have chosen to use a Python Dictionary to store the data, this was a simple task of identifying the Array key and assinging the associated value to a Python Dictionary element as shown.
I do plan to read the directory where the ecowitt data is stored from a variable in the config file, but for now I am using a constant
Another point to note in this program is that Ecowitt populates the directory with a file every minute whereas WeeWX reads the files every 5 minutes, so to handle that I order the files in the directory and delete all but the most recent file, which I then read and parse. This ensures that even if Ecowitt stops sending files for whatever reason there will always be at least 1 file in the directory and the process will succeed.
Again I may make this a bit more sophisticated going forward and do a timestamp check so that old data is not written into the database, but in the meantime the file is available for download here
Finally you will spot that I typecast the data I read into a float before inserting it into the database, this is because the data is stored as data type REAL in the database but when it is read from the text file it is assumed to be a string
pi@raspberrypi:~ $ vi ecowitt.py
import os
import weewx
from weewx.engine import StdService
from stat import S_ISREG, ST_MTIME, ST_MODE
ECOWITT_PATH = '/var/www/html/ecowitt'
class AddEcowittData(StdService):
def __init__(self, engine, config_dict):
# Initialize my superclass first:
super(AddEcowittData, self).__init__(engine, config_dict)
# Bind to any new archive record events:
self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)
def new_archive_record(self, event):
lEcowittFiles = (os.path.join(ECOWITT_PATH, sFiles) for sFiles in os.listdir(ECOWITT_PATH))
lEcowittFiles = ((os.stat(sFullPath), sFullPath) for sFullPath in lEcowittFiles)
lEcowittFiles = ((stat[ST_MTIME], sFullPath) for stat, sFullPath in lEcowittFiles if S_ISREG(stat[ST_MODE]))
sDelFile = ''
for cDate, sFullPath in sorted(lEcowittFiles, reverse=False):
if os.path.exists(sDelFile):
os.remove(sDelFile)
sDelFile = sFullPath
else:
sDelFile = sFullPath
dEcowittData = {}
ecowittData = open(sFullPath, 'r')
for sLineIn in ecowittData:
if "[" in sLineIn:
sKey = sLineIn.split('[', 1)[1].split(']', 1)[0]
sValue = sLineIn.split('=>', 1)[1].strip()
dEcowittData[sKey] = sValue
ecowittData.close()
event.record['extraTemp3'] = float(dEcowittData['tempinf'])
event.record['extraHumid1'] = float(dEcowittData['humidityin'])
event.record['extraTemp2'] = float(dEcowittData['tempf'])
event.record['extraHumid2'] = float(dEcowittData['humidity'])
Put your Python Script into the Correct Directory
WeeWX scans a number of directories looking for scripts and files to run, one of those directories is a user directory where you can put all of your WeeWX extensions and this is where we are going to put this python file. Full documentation on customizing WeeWX can be found here
Copy your new Python script into the bin user directory under your WeeWX home directory
in my case this is /home/weewx
pi@weewx: $ cp ecowitt.py /home/weewx/bin/user/
pi@weewx: $ ls -al /home/weewx/bin/user/
total 48
drwxr-xr-x 3 pi root 4096 Jan 14 11:52 .
drwxr-xr-x 12 pi root 4096 Mar 28 2020 ..
-rw-r--r-- 1 pi root 829 Mar 28 2020 cputemp.py
-rw-r--r-- 1 pi root 4170 Jan 14 11:52 ecowitt.py
-rw-r--r-- 1 pi root 541 Mar 14 2020 extensions.py
-rw-r--r-- 1 pi root 306 Mar 14 2020 __init__.py
drwxr-xr-x 2 pi root 4096 Jan 14 11:52 __pycache__
-rwxr-xr-x 1 pi pi 7327 Mar 28 2020 s3backup.py
-rwxr-xr-x 1 pi pi 11041 Mar 28 2020 s3upload.py
-rwxr-xr-x 1 pi pi 3992 Mar 28 2020 stats.py
Now configure weewx.conf
Before your WeeWX installation will start processing the Ecowitt data, you have to tell WeeWX to run your new Python script.
This is done by modifying the Services section of your weewx configuration file and I know there are some fancy ways of doing this programatically, but I simply edit the file by adding a new entry for my new Python script
The syntax of this new entry is critical and is explained below
data_services = user.ecowitt.AddEcowittData
- data_services: Is an existing service that we use to add data to the database
- user: Is the directory under bin where the new Python script is located
- ecowitt: Is the name of the new Python Script
- AddEcowittData: Is the name of the class within the new Python Script
pi@weewx: $ vi weewx.conf
# This section configures the internal weewx engine.
[Engine]
[[Services]]
# This section specifies the services that should be run. They are
# grouped by type, and the order of services within each group
# determines the order in which the services will be run.
prep_services = weewx.engine.StdTimeSynch
data_services = user.cputemp.AddCpuTemp, user.ecowitt.AddEcowittData
process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate
archive_services = weewx.engine.StdArchive
restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS
report_services = weewx.engine.StdPrint, weewx.engine.StdReport
Set Heading Text
This file also contains the labels that are used as the headings for each graph, and they are defined in the Labels section
Look for the value extraTemp1 and set the text to a value of your choice, I have used CPU Temperature
pi@weewx: $ vi weewx.conf
# The labels to be used for each observation type
[[[Labels]]]
...
# Generic labels, keyed by an observation type.
[[[[Generic]]]]
barometer = Barometer
...
windvec = Wind Vector
extraTemp1 = CPU Temperature
extraTemp2 = Caravan Temperature
extraTemp3 = Lounge Temperature
extraHumid2 = Caravan Humidity
extraHumid1 = Lounge Humidity
Set Quality Control Checks
This file also contains a section for quality control where we can define the units and ranges for the database fields
As we are using some extra fields it is a good idea to set the values in this section
pi@weewx: $ vi weewx.conf
[StdQC]
[[MinMax]]
barometer = 26, 32.5, inHg
pressure = 24, 34.5, inHg
outTemp = -40, 120, degree_F
inTemp = 10, 120, degree_F
outHumidity = 0, 100
inHumidity = 0, 100
windSpeed = 0, 120, mile_per_hour
rain = 0, 10, inch
extraTemp1 = 10, 200, degree_F
extraTemp2 = -40, 120, degree_F
extraTemp3 = 10, 120, degree_F
extraHumid2 = 0, 100
extraHumid1 = 0, 100
Add The New Temperature Graph
The creation of graphs is controlled in skin.conf and the one is use is the new Seasons file and this is the modification I will be descrbing
As with the other configuartion files I manually edit this configuration file, adding a new entry between %tempin and %rain to generate the new graphs as shown below
Note that we are creating four graphs here, daily, weekly, monthly and yearly
pi@weewx: $ vi skin.conf
[[[dayCaravantemp]]]
[[[[extraTemp2]]]]
[[[dayLoungetemp]]]
[[[[extraTemp3]]]]
[[[weekCaravantemp]]]
[[[[extraTemp2]]]]
[[[weekLoungetemp]]]
[[[[extraTemp3]]]]
[[[monthCaravantemp]]]
[[[[extraTemp2]]]]
[[[monthLoungetemp]]]
[[[[extraTemp3]]]]
[[[yearCaravantemp]]]
[[[[extraTemp2]]]]
[[[yearLoungetemp]]]
[[[[extraTemp3]]]]
Finally Display the new Graphs
Creating the graph is only half the battle, you also have to tell WeeWX to display the new graph, I have chosen to display my graph in the Maion section and that is the example that I will be using below
First you need to locate the template that is used to generate the WEB pages, for the Sensor Status this is index.html.tmpl
As with the previous changes, formatting is critical and in this file you are telling the WeeWX engine which files to display and the name you use in this file must match the name you generated in the skin.conf file, in my case this is daytemp, weektemp, monthtemp and yeartemp
pi@weewx: $ vi index.html.tmpl
<div id="history_day" class="plot_container">
<img src="daytempdew.png" alt="$obs.label.outTemp" />
<img src="dayCaravantemp.png" alt="$obs.label.extraTemp2" />
<img src="daytempin.png" alt="$obs.label.inTemp" />
<img src="dayLoungetemp.png" alt="$obs.label.extraTemp3" />
<div id="history_week" class="plot_container" style="display:none">
<img src="weektempdew.png" alt="$obs.label.outTemp" />
<img src="weekCaravantemp.png" alt="$obs.label.extraTemp2" />
<img src="weektempin.png" alt="$obs.label.inTemp" />
<img src="weekLoungetemp.png" alt="$obs.label.extraTemp3" />
<div id="history_month" class="plot_container" style="display:none">
<img src="monthtempdew.png" alt="$obs.label.outTemp" />
<img src="monthCaravantemp.png" alt="$obs.label.extraTemp2" />
<img src="monthtempin.png" alt="$obs.label.inTemp" />
<img src="monthLoungetemp.png" alt="$obs.label.extraTemp3" />
<div id="history_year" class="plot_container" style="display:none">
<img src="yeartempdew.png" alt="$obs.label.outTemp" />
<img src="yearCaravantemp.png" alt="$obs.label.extraTemp2" />
<img src="yeartempin.png" alt="$obs.label.inTemp" />
<img src="yearLoungetemp.png" alt="$obs.label.extraTemp3" />
I also chose to display the data in the HiLo section, and this is done by making these changes to hilo.inc
pi@weewx: $ vi index.html.tmpl
<tr>
<td class="label">$obs.label.extraTemp2</td>
#for $archive in $archive_data
<td class="data new_row hilo_$archive[0]">
<span title="$archive[1].extraTemp2.maxtime">
$archive[1].extraTemp2.max.format(add_label=False)</span><br/>
<span title="$archive[1].extraTemp2.mintime">
$archive[1].extraTemp2.min.format(add_label=False)</span>
</td>
#end for
<td class="units">$unit.label.extraTemp2</td>
</tr>
And that is it, you should now get a your new graphs on your front page, someting like this