All Postings (95)

2017

Best of 2017

Working Remotely

Computer Science and IT

Taking stock! Physics

Subversive? Physics?

My Philosophy!

Scripts Beget Scripts

2016

Theoretical Physics. A Hobby.

Self-Referential Poetry

Silent Online Writing

'Are You Still Doing PKI?'

My Philosophy (?)

Impact of physics on my life

Not much happened in 2015

2015

Unspeakable

Self-Poetry

Farewell Posting ...

Hacking away...

Web Project - Status

We Interrupt ...

Poetry from Poetry

PKI-Status-Update

Life and Work

Definition: 'Subversive'

2014 in Books

Physics Postings

Engineering Postings

True Expert

2014

2014 - a Good Year

Physics or Engineering?

Engineering Links

What Is Art?

Bio

PKI FAQ

Google's Poetic Talents

Certificates and Heat Pumps

Nr. 5: A Mind-Altering Experience

Technet postings

WOP!

Pink Spaceship

radices = Roots!

IT Postings

Web Projects

Life, the Universe, and Everything

Uh-oh, No Posting in March

PKI Resources

PKI Issues

Subversive Work

Spam Poetry

A Career 'in Science'

Writing

On the Shoulders of Subversive Giants

Search Term Poetry

Facebook Art

2013 in Books

2013

Explain, Evaluate, Utilize

Technology

About Life-Form Elke Stangl

elkement and This Site

No. 3: Internet Apocalypso

Retrospection

Newsletter Resurrection

2012

For Free

Subversive Yearly Report

Is My Life a Cliché?

Indulging in Cliché

Torture Turning Trivia

Intermittent Netizen

Knowledge Worker...

Profile

Physics on the Fringe

Graduation Speech

The Element is Back!

Offline

Physics Links

2011

Not Funny

Calendar and Magic

Expert

In Need of a Deflector

About to Change

A Nerd's Awakening

For the Sake of Knowledge

2008

Profession Or True Calling?

No. 2: On Self-Reference

I Have No Clue About Art

Netizen

2007

The End

No. 1: On Subversion at Large

2005

Emergency Exit

Modern Networker

2004

The Scary Part

Exploring the Work Space

2003

Instead of a CV

Favorite Books

2002

Elke was here

Postings tagged with 'Log File', listed in descending order by creation date.

Sometimes I wonder why I had created a Tech category separate from an IT category. The two of them are interrelated closely as my recent Wordpress blog post on my so-called Data Kraken had demonstrated.

I call myself the Theoretical Department of our engineering consultancy because I am mainly in charge of software development, simulations, and data analysis – related to measurement data for our heat pump system (and those of our clients).

But there is one big difference between what I call 'IT-only projects' (like my PKI-related services) or engineering projects that also involve software: 'IT' is my tag for providing software-related consulting or software engineering related to somebody else's IT system – a system whose requirements are defined by somebody else. My engineering software is built according to my own requirements. My 'Tech' projects, IT-centered as they may seem, are not primarily about IT: They are about systems using, storing, and transferring energy. IT is just a tool I use to get the job done.

All things I had ever done as an IT professional turn out to be useful, and I am learning something new nearly every day – when thinking about 'energy'. Heating systems today are part of what is called Internet of Things – so IT security is also an important aspect to consider. In 2015 I used this website to finally transition to .NET (… finally, from ASP ?), and as a spin-off I also re-developed the numerical simulations for our heat pump system in .NET – representing every component as on object. 2014 I migrated our initially only Excel-based data analysis to SQL Server, and I have improved my 'Data Kraken framework' since then, adding visualization by automated Excel plots etc.

I still work for some select 'IT-only' clients - and it seems my 'IT articles' here just constitute a series of updates about the exact extent to which I still do PKI. If the occasional data analysis question comes up, any SQL, Excel, or .NET skills might come in handy in my IT projects - like querying a certification authority's database, or using a semi-automated Excel sheet to create a Certificate Policy Statement, following the RFC. But I don't advertise myself as a SQL etc. expert; I rather think I returned to where I came from, many years ago:

When I worked as an IT consultant, I had been asked over and over: How does a physicist end up in IT? There are very different reasons: The obvious one is that as a physicist you might have picked some programming experience. I had indeed contributed to the (mess of patchy 'local-community-developed') software for automating the measurement of electrical resistance of superconducting thin films many years ago, but this was not the main reason. I was an experimental physicist so I can't claim that my work was immensely mathematical or computational (and my job as 'implemented applied cryptography' via Public Key Infrastructures was not either). The main analogy is that IT systems of sufficient complexity are as unpredictable as an experimental setup governed by lots of parameters, some of which you have not identified yet – as was the manufacturing of thin films by laser ablation. I was simply patient, perseverant, and good at troubleshooting by navigating a hyperspace of options what might have gone wrong.

This might be either boring or frustrating for non-geeks. But I believe the grunt work of maintaining and fixing software is rewarding if this is an auxiliary task, done to support the 'actual' system of interest. Mine are heat pump systems, power meters, photovoltaic generators and the like. I want to understand and optimize them and so I am willing to learn new programming languages and spend hours on troubleshooting bugs with software vendors' updates. Just as back then I learn the bare minimum of Turbo Pascal to develop software for low temperature measurements.

In 2017 I am going to focus on maintaining (and bug fixing ?) Data Kraken und ich will work on making usage and 'visualization' of the numerical simulation more and more similar to Data Kraken.

Currently, Data Kraken has the following main features:

  • Documentation of the sensors and log files for different loggers (Heat pump / UVR16x2, smart meter, PV…) in an Access database - a small proto-kraken per installed system.
  • Documentation of changes to sensors and log files, such as: Shuffled columns in files, modified naming conventions for files, new or replaced sensors. For example, the formerly manual reading off of the surface level of water in the water/ice tank has been replaced with an automated measurement in 2016. So the input value for calculating ice volume moved to a column in a different log file, and was measured in different time intervals.
  • A Powershell script grabs all log files from their source locations, and changes date formats, decimal commas and line breaks. (I found this to be more performant than manipulating every line later after the import to SQL Server).
  • The Powershell script then creates an updated set of SQL scripts – one set of scripts and one SQL database for each installation / each client. For example, the CREATE TABLE or ALTER TABLE commands are created based on the Access documentation of measured values and their change log.
  • SQL scripts create or add SQL Server database fields, import only the files containing data points not imported yet, and import their data to a staging table. Each SQL database can thus always be re-created from scratch – from CSV log files and the meta documentation (Access).
  • Error values are modified or deleted from the staging table, as defined before in the Access database (and such in a SQL script): For example vendor-defined error values for not connected sensors (as 9999) are set to NULL or whole rows of values are deleted if the system was e.g. subject to maintenance according to other system's documentation.
  • Finally, the most important script is run: The one that does the actual calculation of e.g. average brine temperature, energy harvested by PV panels or the solar / air collector by day, or daily performance factors of the heat pump. The script needs several levels of SQL views – all of which are re-created by the script.
  • Microsoft Excel is used as a front-end to show values from tables with calculation results. One Excel-formula only simple table allows for browsing through values, and picking daily, monthly, yearly, or seasonal numbers.
  • Excel plots are automated with respect to the fields (columns) and to start and end date. Existing plots can be copied (also from other workbook), then documented in a table. The documentation table can then be modified and is used as input. Color and line widths are still tweaked manually.

Weird as this setup sounds, it allowed me to develop and change the solution just in the right way – installation by installation, e.g. by testing the changes to log files after the control unit's firmware for one specific installation first.

Data Kraken - front-end

Personal website of Elke Stangl, Zagersdorf, Austria, c/o punktwissen.
elkement [at] subversiv [dot] at. Contact