Archive of postings for 2017, listed in descending order by creation date. All Postings shown.
I did not have ambitions 2017. It should have been a year of taking stock -
and it was, in a good way.
time-travelled and re-lived some history of software engineering, and
finally learned basics of computer science. This was philosophical delight,
but also useful and necessary: I was able to boost the performance of my
simulations (above a level of what was, maybe, embarrassingly slow).
- I tinkered a lot with
numerical simulations of our heat pump system. Main thing I learned: The
more modern the building, the more you'd need to simulate humans' behavior,
rather than physics or control logic.
- Reverse engineering and troubleshooting is what finally connects all the
fields of science and engineering I love: Troubleshooting, ferreting out
hidden causes and effects in hydraulics feel the same as
sniffing and debugging software and networking protocols.
- Theoretical physics reading: I returned to classical thermodynamics and
statistical mechanics; I find it fascinating and beautiful in its own
right, even if only at the pre-1960s level. I took stock of my
writing on heat transport - and I am happy I can actually really use
physics on a daily basis, in down-to-earth engineering projects.
- I was thinking about automation, standardization, and big social media
platforms. I struggled with this blog post about the
future of small business for a long time, but optimism won. I might
frame this even more positively today: There is a place for artisanal
service delivery despite or because of Everything Being Offered As A Service
by Omniscient Data Krakens.
- My blog turned 5 in spring, and I allowed myself to
return to a more philosophical blogging style (briefly). Otherwise, I
finally and subconsciously made the elkement.blog my main resources of
technical content - or at least content related to my professional domain,
and content edited for clarity and entertainment. Whereas on
elkement.subversiv.at I let my stream of consciousness flow. It seems that
the pattern that finally emerges is: elkement.blog = elkement's tech /
science magazine and platform for personal research news, with an ever
growing focus on fields I have training in and daily practical exposure to.
elkement.subversiv.at gravitates against the same focus, but I allow myself
to focus on my personal perspective only. So here you find 'what I am doing
with [insert: heat pumps, security,...]', over there you find the useful
content as such.
- Tomato harvest was great. I tried to grow late varieties - like Ox Heart
- directly outside, and it worked.
- Dinosaur Kale tastes good. And it is able to recover from a at attack of
a bug that targets kale (and radishes' seed capsules). Don't try to keep
seeds of radishes in the land of canola.
This website is an old-school non-interactive site. My blog technically
isn't, but looks like one now, for the lack of visible comments. However,
messages have reached over covert 1:1 channels, so I do now that there is a
small but sincerely interested group of readers. I thank you all for reading my
Many years ago, The Web – which has its own category on my website here –
was an experimental playground for me. You might have guessed so, just
checking out the URL of this post.
Technologies and protocols once used for displaying static websites have
been repurposed, and HTTP(s) became the so-called Universal Firewall Bypass
protocol. We synchronize files with Dropbox or offline-cache or mailboxes.
Applications like Teamviewer or the signals from our Things (as in Internet
Of Things) poke controlled holes into our firewalls so that they are
somewhat accessible from the outside.
I have written about all of that at length elsewhere – about the
insecurity of the Internet of Things and about
Data Krakens dominating small businesses. I have had mixed feelings
about the evolution of The Web. But there is one absolutely positive
outcome: That HTTP(s) (mis-)use connection magic enables me to work in a way
I would have never envisaged 25 years ago – at the time when my most
important ‘files’ were still contained in physical folders.
I am able to work nearly remote-only, not only in IT projects. About 10
years ago I was a consultant in information security. We worked from ‘home
office’, too, although company culture often dictated that there had to be
meetings in real life. Today, I still support some long-term IT security
clients, but mainly via remote and/or asynchronous channels. When we started
our experimental heat pump side-business several years ago, my standard joke
was: Someday we will work in heat pump projects the way we work in IT
projects. And the joke became true – it actually became the default way of
working, even for clients that are within geographical reach, like a 50-70km
This list on our website explains the steps / stages of such a project –
but it’s hard to convey the spirit of a remote project properly. It sounds
way too serious. On our German blog we feature
verbatim hilarious quotes of a client / ice storage heat pump system
self-builder – translation could never do it justice.
Working remotely seems to be about technology: We need to have the tools
we have today to communicate, exchange information, to monitor and manage
systems over the internet. But it is more about culture. In IT, such tools
have already been available for a long time, yet some corporations insisted
on ‘face showing rituals’. Notably, during the economic crisis of 2008/2009
many companies worked hard to keep travel costs low and resorted to working
remotely – and later never reverted to face showing mode.
Successful remote communication is based on the skill of asynchronous
communications, e.g. on processing more than the first three lines of an
e-mail, but replying thoughtfully in nested threads. My anecdotal evidence
tells me that our typical heat pump clients have that skill – tech-savvy
geeks whose day jobs are usually tech- / IT- / engineering-related .
You need to keep politics out. As soon as that infamous ‘non-verbal
clues’ become important, remote channels might be too narrow. However, I
wonder if politics can ever be tamed properly even with heavy face showing.
My pragmatic solution is to focus on simple ‘structures of command’ – work
with one single accountable client who is in charge for his/her project and
has skin in the game. Only if you need to intermediate between ‘team
members’ and listen to ‘different sides’ you get into troubles. I have my
share of experiences – like: Clandestine meetings in which project member X
told me they considered to revolut against project manager Y – depending on
my honest opinion of Y.
Many hands-on engineering tasks are gradually being supported by remote
IT tools. I am not a first adopter of such technology – like augmented
reality glasses for engineers in power plants. My icon is an angry dinosaur
for a reason. But even I say, half-jokingly, that someday people might 3D
print our heat exchanger tubes and PVC supporting constructions, instead of
working with our traditional design documents and plans.
So at the end of 2017, I embrace The Web again and my outlook is
positive. It’s like returning to the old days – when
The Cluetrain Manifesto told us that The Internet will kill TV-like ads and
foster communications between human beings – also in business. That may
sound irrational, given the ominous power of online tracking, all for the
sake of advertizing. But anyway: The positive spirit of
remote working pioneers, like Automattic (wordpress.com) is what defines
The Web for me!
I will try to explore my relationship with IT / software / computers / computer science / software engineering or whatever the best term is to describe it. I am in a mode of looking back with content, and making small changes, learning a bit more.
As often, thinking in 'opposites' comes most natural to me:
Self-study versus formal education. The IT and software industry is young and - I believe - had originally been populated by people without a formal training in computer science as this did not yet exist as an academic discipline. The community was open to outsiders with no formal training or unrelated experience. As a former colleague with a psychology background put it: In the old times, anybody who knew how to hold a computer mouse correctly, was suddenly considered an expert.
I absorbed the hacker ethics of demonstrating your skills rather than showing off papers, and I am grateful about the surprisingly easy start I had in the late 1990s. I just put up a sign in a sense, saying Will Do Computers, and people put trust in me.
I am not 'against' formal education though. Today I enjoy catching up on computer science basics by reading classics like Structure and Interpretation of Computer Programs.
Breaking versus building things. I have been accountable for 'systems' for a long time, and I have built stuff that lasted for longer than I expected. Sometimes I feel like a COBOL programmer in the year of 2000.
But I believe what interested me most is always to find out how stuff works - which also involves breaking things. Debugging. Reverse Engineering. Troubleshooting. All this had always been useful when building things, especially when building on top of or interfacing with existing things - often semi-abandoned blackboxes. This reverse engineering mentality is what provided the connection between physics and IT for me in the first place.
It was neither the mathematical underpinnings of physics and computer science, or my alleged training in programming - I had one class Programming for physicists, using FORTRAN. It was the way an experimental physicist watches and debugs a system 'of nature', like: the growth of thin films in a vacuum chamber, from a plasma cloud generated by evaporating a ceramic target bombarded with laser pulses. Which parameter to change to find out what is the root cause or what triggers a system to change its state? How to minimize the steps to trace out the parameter space most efficiently?
Good-enough approach versus perfectionism. 80/20 or maybe 99/1. You never know or need to know anything. I remember the first time I troubleshooted a client's computer problem. I solved it. Despite knowing any details of what was going on. I am sort of embarrassed by my ignorance and proud at the same time when I look back.
In moment like this I felt the contrast between the hands-on / good-enough approach and the perfectionism I applied in my pervious (academic) life. I remember the endless cycles of refinement of academic papers. Prefixing a sentence with Tentatively, we assume,... just to be sure and not too pretentious though I was working in a narrow niche as a specialist.
But then - as a computer consultant - I simply focused on solving a client's problem in a pragmatic way. I had to think on my feet, and find the most efficient way to rule out potential root causes - using whatever approach worked best: Digging deep into a system, clever googling, or asking a colleague in the community (The latter is only an option if you are able to give back someday).
Top-down, bottom-up, or starting somewhere in the middle. I was not a typical computer nerd as a student. I had no computer in high school except a programmable calculator - where you could see one line of a BASIC program at a time. I remember I had fun with implementating of the Simplex algorithm on that device.
However, I was rather a user of systems, until I inherited (parts of) an experimental setup for measuring electrical properties of samples cooled down by liquid nitrogen and helium. I had to append the existing patchwork of software by learning Turbo Pascal on the job.
Later, I moved to the top level of the ladder of abstraction by using *shock, horror* Visual Basic for Applications, ASP, and VBScript. In am only moving down to lower levels now, finally learning C++, getting closer to assembler and thus touching the interface between hardware and software. Which is perhaps where a one should be, as a physicist.
Green-field or renovation (refactoring). I hardly ever had the chance to or wanted to develop something really from scratch. Constraints and tough limiting requirements come with an allure of their own. This applies to anything - from software to building and construction.
So I enjoy systems' archaeology, including things I have originally created myself, but not touched in a while. Again the love for debugging complements the desire to build something.
From a professionals' point of view, this is a great and useful urge to have: Usually not many people enjoy fiddling with the old stuff, painstakingly researching and migrating it. It's the opposite of having a chance to implement the last shiny tool you learned about in school or in your inhouse presentation (if you work for a software vendor).
In awe of the philosophy of fundamentals versus mundane implementation. I blogged about it recently: Joel Spolsky recommended, tongue-in-cheek, to mention that
Structure and Interpretation of Computer Programs brought you to tears - when applying for a job as a software developer.
But indeed: I have hardly attended a class or read a textbook that was at the same time so profoundly and philosophically compelling but also so useful for any programming job I was involved in right now.
Perhaps half of older internet writing reflects my craving for theses philosophical depths versus the hard truth of pragmatism that is required in a real job. At the university I had been offered to work on a project for optimizing something about fluid dynamics related to the manufacturing of plastic window frames. The Horror, after I had read Gödel, Escher, Bach and wanted to decode the universe and solve the most critical problems of humanity via science and technology.
I smile at that now, with hindsight. I found, in a very unspectacular way, that you get passionate about what you are good at and what you know in depth, not the other way round. I was able to possibly reconnect with some of my loftier aspirations, like I could say I Work In Renewable Energy. However, truth is that I simply enjoy the engineering and debugging challenge, and every mundane piece of code refverberates fundamental truths as the ones described in Gödel, Escher, Bach or Structure and Interpretation.
I have been blogging 'seriously' about physics since 2012. My motivation has
a blend of jotting down notes on interesting things I've just found, conveying my decades-old fascination with some phenomena, trying my hands at
popular science writing, and reporting on my own research.
Today I am asking myself - did I learn anything from that on a meta level? To
read myself, I am re-arranging the list of my physics posts and sort them by
topic and sub-topic. The list says it all, I think.
I wanted to write about quantum mechanics, but it seems I was always most
intrigued by classical mechanics, statistical mechanics, and thermodynamics. The
latter has become my true home in physics - which has come as a surprise to
myself. Yes, thermodynamics is my specialization, but years ago I rather figured
that this is my job, and I rather want to follow the latest news on quantum
information and particle theory in my spare time. It turned out that I am more
interested in history of physics and in the evolution of concepts that are now
My recurring meta-topic is that classical mechanics / thermo can be as
interesting, 'geeky' if you wish, mathematically 'weird', and surprising as
fields that seem to be more popular.
This list may remain a static snapshot. I am editing the chronological list
of my physics posts
here on the blog. This list might lseem to lack some of my more applied /
engineering postings, re our heat pump system and data.
These are here.
Thermodynamics and Statistical Mechanics
Concepts and foundations
Random Thoughts on Temperature and Intuition in Thermodynamics
Time evolution of systems in phase space:
On the Relation of Jurassic Park and Alien Jelly Flowing through Hyperspace.
Phase-space in depth:
Hyper-Jelly – Again. Why We Need Hyperspace – Even in Politics.
efficiency, irreversibility, proof by contradiction, paradoxes:
Re-Visiting Carnot’s Theorem.
Mathematics used in statistical
Spheres in a Space with Trillions of Dimensions.
Heat pump basics
Brief explanation, absolute temperature:
Why Do Heat Pumps Pump Energy so Easily?
Coefficient of Performance of
a heat pump:
An Efficiency Greater Than 1?
Cross-check of numbers for a large
heat pump system:
Pumped Heat from the Tunnel
COP versus Performance Factor:
How to Evaluate a Heat Pump’s Performance?
Heat Pump System Data: Three Seasons 2012 – 2015.
Heat conduction, diffusivity, latent heat:
Storage Challenge: High Score!
Heat conduction, heat equation:
Temperature Waves and Geothermal Energy.
Heat diffusion length:
Rowboats, Laser Pulses, and Heat Energy (Boring Title: Dimensional Analysis).
Simple version, daily energy balances:
More Ice? Exploring Spacetime of Climate and Weather.
Heat transport, energy balances. Simulations versus simple energy accounting:
Ice Storage Hierarchy of Needs.
Detailed version: Heat
exchangers, heat equation, 1-minute time slots:
Simulating Peak Ice.
Thermodynamics and energy basics, dimensional analysis
kW and kWh.
No, You Cannot ‘Power Your Home’ by One Hour of Cycling Daily.
Phase transitions, ideal gas law (pressure sensor)
Mr. Bubble Was Confused. A Cliffhanger.
kWp, power, energy, energy flow:
On Photovoltaic Generators and Scattering Cross Sections.
History and inventions
Einstein’s Refrigerator and other inventions:
Einstein and His Patents
Centennial light bulb,
111 Years: A Shining Example of Sustainable Product Development?
Checking 19th century papers:
Peter von Rittinger’s Steam Pump (AKA: The First Heat Pump).
Phase transitions of water, Mpemba effect:
A Sublime Transition.
By an Austrian start-up – pressure gradient created
by centrifugal forces:
And Now for Something Completely Different: Rotation Heat Pump!
(I realize that some of my articles in the
category would also qualify for this sub-category History. For example: I
wrote a - less detailed -
post on Rittinger's steam pump before. But part of the fun with these list
is that you have to take those hard decisions of tagging ...)
Classical Mechanics and Fluid Dynamics
Equations of motion and Lagrangian formalism
Principle of Least Action.
Sniffing the Path (On the Fascination of Classical Mechanics)
Equation of motion, intuition in physics:
Are We All Newtonians?
Motion of a falling slinky spring:
The Falling Slinky and Einstein’s Elevator.
Least Action, again – extended version:
Space Balls, Baywatch and the Geekiness of Classical Mechanics.
From Newton’s Law to Navier-Stokes Equations:
Non-Linear Art. (Should Actually Be: Random Thoughts on Fluid Dynamics).
Estimates related to the physics of scything:
Grim Reaper Does a Back-of-the-Envelope Calculation.
Back-of-the-envelope cross-checks, hydro power:
All Kinds of Turbines.
Torque, forces, precession, nutation.
The Spinning Gyroscope and Intuition in Physics.
way to explain how the gyroscope works:
Intuition and the Magic of the Gyroscope
The Twisted Garden Hose and the Myth of the Toilet Flush:
Coriolis force (2).
Lest We Forget the Pioneer: Ottokar Tumlirz and His Early Demo of the
Physics and geometry
My first (later corrected) proposal of a solution:
Physics / Math Puzzle: Where Is the Center of Mass?
Correcting my earlier proposal:
Revisiting the Enigma of the Intersecting Lines and That Pesky Triangle.
Newton’s geometrical proof of Kepler’s laws:
Mastering Geometry is a Lost Art.
Quantum Mechanics and Quantum Field Theory
Interpretations of quantum mechanics:
Is It Determinism if We Can Calculate Probabilities Exactly?
Quantum Mechanics versus QFT:
Quantum Field Theory or: It’s More Than a Marble Turned into a Wiggly Line.
And Now for Something Completely Different: Quantum Fields!
Summary on QM:
May the Force Field Be with You: Primer on Quantum Mechanics and Why We
Need Quantum Field Theory
Quantization – starting from statistical
On the Relation of Jurassic Park and Alien Jelly Flowing through Hyperspace.
Path integrals and symmetries:
Learning Physics, Metaphors, and Quantum Fields.
Student Friendly Quantum Field Theory.
Electromagnetism and special relativity
Unification of Two Phenomena Well Known.
lose energy when accelerating:
Why Fat Particles Radiate Less.
An alternative way of
How to Introduce Special Relativity (Historical Detour).
Learning General Relativity.
Using physics-like methods in economics and sociology
Networking theory, instabilities:
Theory and Practice of Trying to Combine Physics with Anything
E-Mails and communications:
Using Social Media in Bursts. Is. Just. Normal.
‘Philosophical’: Learning physics, culture
On trying to explain physics without math:
Real Physicists Do Not Read Popular Science Books
Trouble with Physics by Lee Smoli:.
I neither Met Newton nor Einstein
On learning physics.
Stupid Questions and So-Called Intuition.
Wertheim’s Physics on the Fringe
Physics Paradoxers and Outsiders.
Physics as Therapy (1)
In Praise of Textbooks with Tons of Formulas (or: The Joy of
Physics as Therapy (2)
Ploughing Through Theoretical Physics Textbooks Is Therapeutic.
This website shall finally reconnect with its roots – radices.
With the dawn of the new millennium a self-proclaimed Subversive Element has registered a bunch of domains. It was especially fond of radices.net and subversiv.at. Today, all these sites have been re-united and redirected to elkement.subversiv.at. But the site does not deliver on its promising name – I feel it became way too 'professional' recently. Historical content has been filed mostly under Physics (radices) and Art (subversiv). The category life displays some of the matter-antimatter collisions of these two worlds. Which also explains the category of the current article.
The Subversive Site was a Red Padded Cell, with Font Color = White, a so-called creative playground. The Element was aware that 'everybody' could read this but it did not care. The Merger of the sites was inevitable in the end, after a final detour of professionalization – when radices.net suddenly also hosted pages with IT Security links.
I have been a blogger, and I observed the evolution of other blogs: My anecdotal evidence shows that blogs live for about 1-2 years. If they are bound to survive they have to escape the matrix and to overturn their creators. A personal blog or website needs a 'Big' Idea. OK, not really big, but at least a-all-encompassing and abstract enough so that all the authors different threads and lines of thoughts can be silently tied together using this idea's magic glue.
My elkement.blog is relentlessly edited. It was a more philosophical site once, but I aim at following our punktwissen principles now. Articles should be concise, provide value, and perhaps also entertainment. There should be s logical connection between posts and my curated lists should help readers to find something 'useful'.
On the contrary, this site has more or less the same article over and over again – perhaps in disguise and interlaced with technical notes. It is all about my personal keeping the essence of Physics alive and useful for me. Since radices was originally a German-only science and philosophy site, the English version might not reflect this – but in the early articles on elkemental Force (at that time: Theory and Practice of Trying to Combine Just Anything) I recaptured these ideas.
So I do finally accept this – let elkement.subversiv.at have its way. This is elkement's personal site, and its primary topic is How To Learn About Physics And Why This Might Be Useful Or Even Edifying In Very Different Ways.
- Learning physics means to start somewhere in the middle. That's why a first Introductions to Physics lecture is always hard (if the lecturer has some modest mathematical aspirations). You need to look at the same phenomena from different angles, and only after a while – and some work – everything will fall into place. This process and journey of learning is rewarding in itself.
- The more related to mathematical foundations (of physics) a question is, the less googleable the answer is. You can find anecdotes, and examples, science sound-bites for entertainment. Of course you find awesome lecture notes to learn the fundamentals from Feynman Lectures to Landau-Lifshitz – but you need to 'learn' them. In contrary to the mantra of You Just Need to Know Where to Find Something (like: Google for error messages) I believe that really knowing about fundamentals without googling helps a lot with problem solving: You can walk through how a system should work, just using the resources in your head.
- Mathematics purges the brain, and this does not only help with
mathematical problem solving. So I believe that the hackneyed
problem-solving skills of science graduates are real (albeit it is difficult
to assess the self-selecting nature of STEM degrees for people with natural
'analytical' skills). But the caveat it: Years of corporate work, powerpoint
slides, office politics, distractions, pressure to deliver ad hoc can erode
these skills. I have long-term tested different methods to keep physics
knowledge alive and usable - and
learned now that science might even provide some evidence, in a sense.
- I have been in 'cyber security' for a while and I have written lots of gloomy articles about our new smart world of automation and where everything (including heating systems) is turned into cloud-based services. Thoughts on all of this is still work in progress, I am working on internal consistency and unambiguity. I came into the world of IT as an experimental physicist, I was applying my training of troubleshooting complex 'analog systems' to digital systems. Despite the myth of crystal-clear 0s and 1s it was often better to treat them as blackboxes. I lacked the typical computer nerd's / enthusiast's background and started late – playing with Microsoft systems and Office VBA and the like.
In spite of this Treat-as-a-Blackbox approach I like to understand as much as possible about a system. Yes, I know you cannot understand, yet build, a power plant, from knowing how to solve Maxwell's Equations (yet understand or solve issues in cyber security related to such power plants). Nevertheless, if I have the choice to understand something at all, I'd pick Maxwell's Equations.
Since years I am using an (angry) dinosaur as my web and blog logo. The dinosaur is from another era, and sometimes it cannot deal with 'modern' concepts of our 'smart', 'networked' world. But perhaps, it was part of this world for a while in order to overcompensate.
Now the dinosaur is getting more and more confident that its typical dinosaur activities might be more productive and positive than it thought before.
On science and technology
- I believe there is often a simpler, a more low-tech solution to a problem technology is thrown on.
- I sometimes call myself a geek but I don't understand this 'geek' movement of cheering science and technology - without any desire to learn any of the details.
- I prefer to work on seemingly mundane problems that somebody really wants me to solve right now.
- This explains why I discarded inquiries to participate in and profit from governmentally funded research projects.
- Yet, I often find a universe of intriguing puzzles when mulling upon a 'simple' problem.
- Learning about theoretical physics has a mind purging effect: It helps, no matter if I ever need the math directly.
On business and life
- If a business relationship does not work without a written contract, it does also not work well with one.
- Don't follow any advice by strategists and experts, especially if their primary role is to act as consultants and not as doers.
- If somebody has an opinion on something, I judge them on Skin in the Game, hands-on experience, and education - in that order. I keep this in mind when voicing my own opinions.
- I don't pay for leads - I endorse other for free, and I am endorsed for free. Not necessarily on a 1:1 basis.
On the internet
- The greatest internet-powered innovation in the workplace I have encountered is to work remotely.
- I am grateful that I started writing online before there were Likes and Comments. The point of writing online is to hold yourself accountable because others could read this on principle, not because you need feedback.
- The internet sharing paradox: The more information you share for free, the more requests for free information you get. Learning to say No is a key skill.
- No matter how eclectic you think your combination of specialties is - you will find people on the internet featuring the same combination. Just better. It's humbling and this is a good thing.
Sometimes I wonder why I had created a Tech category separate from an IT category. The two of them are interrelated closely as
my recent Wordpress blog post on my so-called Data Kraken had demonstrated.
I call myself the Theoretical Department of our engineering consultancy because I am mainly in charge of software development, simulations, and data analysis – related to measurement data for our heat pump system (and those of our clients).
But there is one big difference between what I call 'IT-only projects' (like my PKI-related services) or engineering projects that also involve software: 'IT' is my tag for providing software-related consulting or software engineering related to somebody else's IT system – a system whose requirements are defined by somebody else. My engineering software is built according to my own requirements. My 'Tech' projects, IT-centered as they may seem, are not primarily about IT: They are about systems using, storing, and transferring energy. IT is just a tool I use to get the job done.
All things I had ever done as an IT professional turn out to be useful, and I am learning something new nearly every day – when thinking about 'energy'. Heating systems today are part of what is called Internet of Things – so IT security is also an important aspect to consider. In 2015 I used this website to finally transition to .NET (… finally, from ASP ?), and as a spin-off I also re-developed the numerical simulations for our heat pump system in .NET – representing every component as on object. 2014 I migrated our initially only Excel-based data analysis to SQL Server, and I have improved my 'Data Kraken framework' since then, adding visualization by automated Excel plots etc.
I still work for some select 'IT-only' clients - and it seems my 'IT articles' here just constitute a series of updates about the exact extent to which I still do PKI. If the occasional data analysis question comes up, any SQL, Excel, or .NET skills might come in handy in my IT projects - like querying a certification authority's database, or using a semi-automated Excel sheet to create a Certificate Policy Statement, following the RFC. But I don't advertise myself as a SQL etc. expert; I rather think I returned to where I came from, many years ago:
When I worked as an IT consultant, I had been asked over and over: How does a physicist end up in IT? There are very different reasons: The obvious one is that as a physicist you might have picked some programming experience. I had indeed contributed to the (mess of patchy 'local-community-developed') software for automating the measurement of electrical resistance of superconducting thin films many years ago, but this was not the main reason. I was an experimental physicist so I can't claim that my work was immensely mathematical or computational (and my job as 'implemented applied cryptography' via Public Key Infrastructures was not either). The main analogy is that IT systems of sufficient complexity are as unpredictable as an experimental setup governed by lots of parameters, some of which you have not identified yet – as was the manufacturing of thin films by laser ablation. I was simply patient, perseverant, and good at troubleshooting by navigating a hyperspace of options what might have gone wrong.
This might be either boring or frustrating for non-geeks. But I believe the grunt work of maintaining and fixing software is rewarding if this is an auxiliary task, done to support the 'actual' system of interest. Mine are heat pump systems, power meters, photovoltaic generators and the like. I want to understand and optimize them and so I am willing to learn new programming languages and spend hours on troubleshooting bugs with software vendors' updates. Just as back then I learn the bare minimum of Turbo Pascal to develop software for low temperature measurements.
In 2017 I am going to focus on maintaining (and bug fixing ?) Data Kraken und ich will work on making usage and 'visualization' of the numerical simulation more and more similar to Data Kraken.
Currently, Data Kraken has the following main features:
- Documentation of the sensors and log files for different loggers (Heat pump / UVR16x2, smart meter, PV…) in an Access database - a small proto-kraken per installed system.
- Documentation of changes to sensors and log files, such as: Shuffled columns in files, modified naming conventions for files, new or replaced sensors. For example, the formerly manual reading off of the surface level of water in the water/ice tank has been replaced with an automated measurement in 2016. So the input value for calculating ice volume moved to a column in a different log file, and was measured in different time intervals.
- A Powershell script grabs all log files from their source locations, and changes date formats, decimal commas and line breaks. (I found this to be more performant than manipulating every line later after the import to SQL Server).
- The Powershell script then creates an updated set of SQL scripts – one set of scripts and one SQL database for each installation / each client. For example, the CREATE TABLE or ALTER TABLE commands are created based on the Access documentation of measured values and their change log.
- SQL scripts create or add SQL Server database fields, import only the files containing data points not imported yet, and import their data to a staging table. Each SQL database can thus always be re-created from scratch – from CSV log files and the meta documentation (Access).
- Error values are modified or deleted from the staging table, as defined before in the Access database (and such in a SQL script): For example vendor-defined error values for not connected sensors (as 9999) are set to NULL or whole rows of values are deleted if the system was e.g. subject to maintenance according to other system's documentation.
- Finally, the most important script is run: The one that does the actual calculation of e.g. average brine temperature, energy harvested by PV panels or the solar / air collector by day, or daily performance factors of the heat pump. The script needs several levels of SQL views – all of which are re-created by the script.
- Microsoft Excel is used as a front-end to show values from tables with calculation results. One Excel-formula only simple table allows for browsing through values, and picking daily, monthly, yearly, or seasonal numbers.
- Excel plots are automated with respect to the fields (columns) and to start and end date. Existing plots can be copied (also from other workbook), then documented in a table. The documentation table can then be modified and is used as input. Color and line widths are still tweaked manually.
Weird as this setup sounds, it allowed me to develop and change the solution just in the right way – installation by installation, e.g. by testing the changes to log files after the control unit's firmware for one specific installation first.