... on a pentesting platform. that became my main 'social network'!
It feels like the natural progression from my walking down the stack: In the last year I re-lived my history of a physicist in IT or an IT security specialist trained as a physicist. I investigated the security of embedded systems and sniffed network traffic - mostly related to monitoring and control of physical devices for 'generating' or storing energy.
I wanted to fill in gaps of knowledge, I turned to classic introductions to computer science, and I caught up on C/C++ and Python. But trying to hack systems is still another kind of skill: I had been a 'defender' for many years, explaining to others how to secure their systems, but I lacked the skills of an attacker.
After I had dabbled in forensics of unknown files and in using automated testing tools with modest success, I decided I want to learn this craft thoroughly. Or was it? Maybe I just want to play and see how far I can get. It was a surprise that I was actually able to hack the entry challenge for that pentesting platform. Fast-forward: I had hacked more than 80% of the active boxes.
My experiences there are both very humbling and very gratifying. Sometimes I struggle with even getting an exploit tool to run as I lack some basic knowledge of compile switches. But sometimes I discover I can leverage some things I didn't even realize consciously or ancient things buried deep in my memory. Who knew that ASP and VBScript would ever be useful again? And my preferences of Python and C++ (for non-destructive purposes) feels eerie now - I could not have picked the languages for my exploit tools better! My adventures with learning SQL Server a few years ago also come in handy, and what I considered my most unprofessional hacks turned out to be most useful: Stringing together 'applications' from scripts and compiles code in different languages, burying one into the other, not being afraid of loads of different quotes embracing each other. As a side effect, I am also more daring when it comes to my non-malicious code now: I have no problems any more to state publicly that I write an application in C# that adds VBA macros to Excel and executes them!
My immersion in this addictive platform also told me something about my learning preferences ... again. I had known it but it was not that explicit: I want to learn from solving problems. That was my intuitive answer once, when colleague had asked how I make myself familiar with new technologies, a freshly released operating system at that time. I replied that I try to solve one specific problem on that new system (involving X.509 certificates then) - and then expand my knowledge from there. I have pontificated about my love of reading textbooks and immersing myself in abstract theory, and this is not a contradiction: Hadn't I ploughed through the later chapters of Structure and Interpretation of Computer Programs - the ingenious explanation how compilers and assembly works - I might not enjoy my attempts to create buffer overflows that much. Which is a topic I need much much more reading and playing with, by the way.
I know am saying the same things again and again and again - here, on my blog, and on social media. It seems my websites have run their course for the time being - I am not actively trying to search for new content to create, and I feel like writing articles that flow naturally, rather than writing semi-scholarly papers with code and data. So I am leaving this article here, on the site that nobody reads, as a hidden away note maybe.
Recently I've changed my story at some social profiles again - to this:
Specializing in: Control systems, software development for measurement data analysis, IT security, troubleshooting and reverse engineering systems with physical (hydraulic) and software (control) components.
I am running a small engineering consultancy together with my husband. We are both physicists, and we focus on designing, programming, and troubleshooting control systems for heating / solar systems, especially heat pump systems with a combination of uncommon heat sources and custom control. For more than 10 years I have implemented, reviewed, and troubleshooted public key infrastructures, and I still do this for some long-term clients.
In contrast to this blog, this site here is more of an extended profile / About Me page. It is my hand-crafted whoami machine.
I think about my exploration of layers of software. tl;dr: I am gradually moving down / back to the lower levels of software, the ones closer to hardware, electronics, control, field bus systems etc.
I've started out learning about micro-controllers in electronics class as a physics student. Then I programmed sensors and actuators for measuring the low-temperature electrical properties of superconductors as a staff scientist at the university (in Turbo Pascal). Yet I jumped up to the top of the software stack and switched to Microsoft scripting languages: VBA, VBScript, ASP when I went 'from research to IT'. Even the first version of my numerical simulation for our heat pump system was an Excel spreadsheet, then a VBA application using spreadsheets.
It seems I needed to trade 'IT' again officially for 'renewable energies' to be motivated to move down the stack again. When I was a non-traditional 'post-graduate' student in in energy engineering I was always been the 'Excel programmer' in group projects. Buth then I went down rabbit holes: Learning SQL Server and Transact-SQL for analyzing our measurement data. Re-writing the simulation software, now based on Visual Basic .NET, for the first time using a true object-oriented design. To get ready for this, I had re-written this website from scratch in .NET before. My so-called Data Kraken uses a combination of Powershell and SQL scripts today.
I finally learned to utilize all my processors in my simulation, and I fixed lots of performance issues. I read Joel on Software cover to cover to re-live the period I 'was in IT' and to catch up on fundamentals. He pointed me to Structure and Interpretation of Computer Programs which I consider the single best ever lecture / course I've ever 'attended'. It is both so deep and philosophical, and at the same time so useful: My simulations became faster by a large factor.
And all the time, I did reverse engineering and debugging. I think I have done this ever since, but always at the level I understood software at the time. Of all the tasks I had as an IT Security / Public Key Infrastructure consultant, troubleshooting weird issues with X.509 certificates was maybe the best one: Digging deep into network traces, reading up on RFCs. Every time I was theoretically only a user of software and services, I ended up debugging in detail - like using Wireshark to track down a weird compatibility issue between my e-mail client and a mail server, when just trying to sign my invocies via a digital signature solution using SMTP.
Then I finally learned C and C++, and I read about Assembly and the art of reverse engineering and malware analysis - to really appreciate the final chapters of SICP, about the self-referential wonders of compilers and interpreters.
Trying to visualize the stack and what happens to the registers, I picked up a very old book - the one I used decades ago in my electronics class - and I jumped into the chapter about micro-controllers. And then it hit me: Those fundamentals, they have not changed much. Yes, different processors have different instruction sets and you might have 8bit, 16bit, or 32bit. But the explanation about the stack, and how to return from a function - this has always been an eternal truth since that electronics book and SICP had been released.
All falls into place: Understanding C is really the pre-requisite for understanding field bus communications, and that is what control units use. Debugging skills are essential when dealing with abandoned engineering software from the stone age.
So I finally found the most logical connection between physics and IT, the place to be as a physicist in IT or in engineering or whatever.
Onword to Python!
I will try to explore my relationship with IT / software / computers / computer science / software engineering or whatever the best term is to describe it. I am in a mode of looking back with content, and making small changes, learning a bit more.
As often, thinking in 'opposites' comes most natural to me:
Self-study versus formal education. The IT and software industry is young and - I believe - had originally been populated by people without a formal training in computer science as this did not yet exist as an academic discipline. The community was open to outsiders with no formal training or unrelated experience. As a former colleague with a psychology background put it: In the old times, anybody who knew how to hold a computer mouse correctly, was suddenly considered an expert.
I absorbed the hacker ethics of demonstrating your skills rather than showing off papers, and I am grateful about the surprisingly easy start I had in the late 1990s. I just put up a sign in a sense, saying Will Do Computers, and people put trust in me.
I am not 'against' formal education though. Today I enjoy catching up on computer science basics by reading classics like Structure and Interpretation of Computer Programs.
Breaking versus building things. I have been accountable for 'systems' for a long time, and I have built stuff that lasted for longer than I expected. Sometimes I feel like a COBOL programmer in the year of 2000.
But I believe what interested me most is always to find out how stuff works - which also involves breaking things. Debugging. Reverse Engineering. Troubleshooting. All this had always been useful when building things, especially when building on top of or interfacing with existing things - often semi-abandoned blackboxes. This reverse engineering mentality is what provided the connection between physics and IT for me in the first place.
It was neither the mathematical underpinnings of physics and computer science, or my alleged training in programming - I had one class Programming for physicists, using FORTRAN. It was the way an experimental physicist watches and debugs a system 'of nature', like: the growth of thin films in a vacuum chamber, from a plasma cloud generated by evaporating a ceramic target bombarded with laser pulses. Which parameter to change to find out what is the root cause or what triggers a system to change its state? How to minimize the steps to trace out the parameter space most efficiently?
Good-enough approach versus perfectionism. 80/20 or maybe 99/1. You never know or need to know anything. I remember the first time I troubleshooted a client's computer problem. I solved it. Despite knowing any details of what was going on. I am sort of embarrassed by my ignorance and proud at the same time when I look back.
In moment like this I felt the contrast between the hands-on / good-enough approach and the perfectionism I applied in my pervious (academic) life. I remember the endless cycles of refinement of academic papers. Prefixing a sentence with Tentatively, we assume,... just to be sure and not too pretentious though I was working in a narrow niche as a specialist.
But then - as a computer consultant - I simply focused on solving a client's problem in a pragmatic way. I had to think on my feet, and find the most efficient way to rule out potential root causes - using whatever approach worked best: Digging deep into a system, clever googling, or asking a colleague in the community (The latter is only an option if you are able to give back someday).
Top-down, bottom-up, or starting somewhere in the middle. I was not a typical computer nerd as a student. I had no computer in high school except a programmable calculator - where you could see one line of a BASIC program at a time. I remember I had fun with implementating of the Simplex algorithm on that device.
However, I was rather a user of systems, until I inherited (parts of) an experimental setup for measuring electrical properties of samples cooled down by liquid nitrogen and helium. I had to append the existing patchwork of software by learning Turbo Pascal on the job.
Later, I moved to the top level of the ladder of abstraction by using *shock, horror* Visual Basic for Applications, ASP, and VBScript. In am only moving down to lower levels now, finally learning C++, getting closer to assembler and thus touching the interface between hardware and software. Which is perhaps where a one should be, as a physicist.
Green-field or renovation (refactoring). I hardly ever had the chance to or wanted to develop something really from scratch. Constraints and tough limiting requirements come with an allure of their own. This applies to anything - from software to building and construction.
So I enjoy systems' archaeology, including things I have originally created myself, but not touched in a while. Again the love for debugging complements the desire to build something.
From a professionals' point of view, this is a great and useful urge to have: Usually not many people enjoy fiddling with the old stuff, painstakingly researching and migrating it. It's the opposite of having a chance to implement the last shiny tool you learned about in school or in your inhouse presentation (if you work for a software vendor).
In awe of the philosophy of fundamentals versus mundane implementation. I blogged about it recently: Joel Spolsky recommended, tongue-in-cheek, to mention that Structure and Interpretation of Computer Programs brought you to tears - when applying for a job as a software developer.
But indeed: I have hardly attended a class or read a textbook that was at the same time so profoundly and philosophically compelling but also so useful for any programming job I was involved in right now.
Perhaps half of older internet writing reflects my craving for theses philosophical depths versus the hard truth of pragmatism that is required in a real job. At the university I had been offered to work on a project for optimizing something about fluid dynamics related to the manufacturing of plastic window frames. The Horror, after I had read Gödel, Escher, Bach and wanted to decode the universe and solve the most critical problems of humanity via science and technology.
I smile at that now, with hindsight. I found, in a very unspectacular way, that you get passionate about what you are good at and what you know in depth, not the other way round. I was able to possibly reconnect with some of my loftier aspirations, like I could say I Work In Renewable Energy. However, truth is that I simply enjoy the engineering and debugging challenge, and every mundane piece of code refverberates fundamental truths as the ones described in Gödel, Escher, Bach or Structure and Interpretation.
Sometimes I wonder why I had created a Tech category separate from an IT category. The two of them are interrelated closely as my recent Wordpress blog post on my so-called Data Kraken had demonstrated.
I call myself the Theoretical Department of our engineering consultancy because I am mainly in charge of software development, simulations, and data analysis – related to measurement data for our heat pump system (and those of our clients).
But there is one big difference between what I call 'IT-only projects' (like my PKI-related services) or engineering projects that also involve software: 'IT' is my tag for providing software-related consulting or software engineering related to somebody else's IT system – a system whose requirements are defined by somebody else. My engineering software is built according to my own requirements. My 'Tech' projects, IT-centered as they may seem, are not primarily about IT: They are about systems using, storing, and transferring energy. IT is just a tool I use to get the job done.
All things I had ever done as an IT professional turn out to be useful, and I am learning something new nearly every day – when thinking about 'energy'. Heating systems today are part of what is called Internet of Things – so IT security is also an important aspect to consider. In 2015 I used this website to finally transition to .NET (… finally, from ASP ?), and as a spin-off I also re-developed the numerical simulations for our heat pump system in .NET – representing every component as on object. 2014 I migrated our initially only Excel-based data analysis to SQL Server, and I have improved my 'Data Kraken framework' since then, adding visualization by automated Excel plots etc.
I still work for some select 'IT-only' clients - and it seems my 'IT articles' here just constitute a series of updates about the exact extent to which I still do PKI. If the occasional data analysis question comes up, any SQL, Excel, or .NET skills might come in handy in my IT projects - like querying a certification authority's database, or using a semi-automated Excel sheet to create a Certificate Policy Statement, following the RFC. But I don't advertise myself as a SQL etc. expert; I rather think I returned to where I came from, many years ago:
When I worked as an IT consultant, I had been asked over and over: How does a physicist end up in IT? There are very different reasons: The obvious one is that as a physicist you might have picked some programming experience. I had indeed contributed to the (mess of patchy 'local-community-developed') software for automating the measurement of electrical resistance of superconducting thin films many years ago, but this was not the main reason. I was an experimental physicist so I can't claim that my work was immensely mathematical or computational (and my job as 'implemented applied cryptography' via Public Key Infrastructures was not either). The main analogy is that IT systems of sufficient complexity are as unpredictable as an experimental setup governed by lots of parameters, some of which you have not identified yet – as was the manufacturing of thin films by laser ablation. I was simply patient, perseverant, and good at troubleshooting by navigating a hyperspace of options what might have gone wrong.
This might be either boring or frustrating for non-geeks. But I believe the grunt work of maintaining and fixing software is rewarding if this is an auxiliary task, done to support the 'actual' system of interest. Mine are heat pump systems, power meters, photovoltaic generators and the like. I want to understand and optimize them and so I am willing to learn new programming languages and spend hours on troubleshooting bugs with software vendors' updates. Just as back then I learn the bare minimum of Turbo Pascal to develop software for low temperature measurements.
In 2017 I am going to focus on maintaining (and bug fixing ?) Data Kraken und ich will work on making usage and 'visualization' of the numerical simulation more and more similar to Data Kraken.
Currently, Data Kraken has the following main features:
- Documentation of the sensors and log files for different loggers (Heat pump / UVR16x2, smart meter, PV…) in an Access database - a small proto-kraken per installed system.
- Documentation of changes to sensors and log files, such as: Shuffled columns in files, modified naming conventions for files, new or replaced sensors. For example, the formerly manual reading off of the surface level of water in the water/ice tank has been replaced with an automated measurement in 2016. So the input value for calculating ice volume moved to a column in a different log file, and was measured in different time intervals.
- A Powershell script grabs all log files from their source locations, and changes date formats, decimal commas and line breaks. (I found this to be more performant than manipulating every line later after the import to SQL Server).
- The Powershell script then creates an updated set of SQL scripts – one set of scripts and one SQL database for each installation / each client. For example, the CREATE TABLE or ALTER TABLE commands are created based on the Access documentation of measured values and their change log.
- SQL scripts create or add SQL Server database fields, import only the files containing data points not imported yet, and import their data to a staging table. Each SQL database can thus always be re-created from scratch – from CSV log files and the meta documentation (Access).
- Error values are modified or deleted from the staging table, as defined before in the Access database (and such in a SQL script): For example vendor-defined error values for not connected sensors (as 9999) are set to NULL or whole rows of values are deleted if the system was e.g. subject to maintenance according to other system's documentation.
- Finally, the most important script is run: The one that does the actual calculation of e.g. average brine temperature, energy harvested by PV panels or the solar / air collector by day, or daily performance factors of the heat pump. The script needs several levels of SQL views – all of which are re-created by the script.
- Microsoft Excel is used as a front-end to show values from tables with calculation results. One Excel-formula only simple table allows for browsing through values, and picking daily, monthly, yearly, or seasonal numbers.
- Excel plots are automated with respect to the fields (columns) and to start and end date. Existing plots can be copied (also from other workbook), then documented in a table. The documentation table can then be modified and is used as input. Color and line widths are still tweaked manually.
Weird as this setup sounds, it allowed me to develop and change the solution just in the right way – installation by installation, e.g. by testing the changes to log files after the control unit's firmware for one specific installation first.
... and first post published to the new site, live and public now :-)
For a short time, the old sites are still available in parallel to the new site.
Looking back, I mainly struggled with:
- My flat-file database - accessing content and all meta information stored in text files, using standards SQL queries.
- Redirect strategy: Existing loads of redirects, temporary ones, permanent 301 ones, nice URLs without physical files...
- Migration of the actual content, uniting what was separated in different sources - asp files, RSS feed, CSV file databases
See also my latest blog post. Which also contains the expected meta-musings on The Web.
Lest we not forget - these were the old sites:
In the past weeks since the last update I've added the following features:
- XML sitemap including English and German posts - URLs and last changed date.
- Make yearly archive URLs 'hackable', thus using just /[lang]/[yyyy] as archive URL.
- Population of meta tags, using also open graph tags.
- Adding 'breadcrumb' / 'where am I' information by highlighting the item just clicked in the menu and side bars: Current category, current post, current tag.
- Assign an optional image to a post via related attributes: Image source, image size or full image tage (for embedding Wikimedia images plus copyright information). If an image should be displayed, but no source is given, add a standard image.
- Display the image automatically on the bottom of the post and use it in the open graph image tag, to be used as a preview image. Calculate height and size from the image's physical size and intended width.
- Create thumbnails of these images, to be shown in the list of posts in the category pages.
- Store all global configuration settings such as tagline in a config file that uses the same [name:] [value] parsing logic as content files.
- Migrate all existing posts on the sites e-stangl.at, radices.net, and subversiv.at, and keep track of where the content came from. (One former .asp page contained one or more 'posts').
- Use one default.aspx for all applications, differences depend on the app name. Example: Don't show post archive for the business page, but show latest posts from Wordpress blog feed instead.
- Clean old content: Replace relative references (../) by absolute ones, replace CSS classes in tags. Move meta infos from content to new file attributes.
Web Server Settings and DNS
- Tested the IIS URL rewrite module with a key map, to be created from Excel documentation. In case of issues with rewriting: Fall back to redirecting in a main ASP file.
- Configure new host names and subdomains in DNS as primary URLs of the new applications. Add new host names for testing to reflect the already existing redirects plus the migration redirects plus the future standard redirects.
- Modify the existing main default.asp, global.asa, and main asp script creating all pages to work with the new redirects (some duplicate code in asp and .net could not be avoided)
- Host name determines application name: One main host name for each (of the 3-4) application. I will use a subdomain of subversiv.at as my new primary host.
- Check if the application has been migrated, as per config parameters. If not the existing redirect logic and existing asp code kicks in - which sends the user to a subfolder depending on host name. This is for historical reasons as I had only one virtual web host in the old times, so e.g. e-stangl.at/ redirected to e-stangl.at/e/
- If the app was migrated, redirect all attempts to use a 'secondary' host to the new one. So e.g. accessing e-stangl.at will be recognized as calling the elkement app and redirect to my new primary name.
- Configuring the application as 'migrated' does not yet redirect any attempt to access one of the old articles. I will have to turn on my rewrite map or code for that.
- Complete all features for all applications before taking 'elkement'
- Feed parser for punktwissen,
- 'image database' for z-village (using small posts with images effectively as entries in a table of images), add an option to show the large version of the image inline.
- Maybe: Ordering of posts in category by changed date, not by created date.
- Limit number of posts on main page and on tag's pages, number = global parameter.
- Replace internal relative URLs to pages in the same virtual directory by absolute ones.
- Maybe: Replace parent path (../) URLs in old code, to turn Parent Path in the ASP config off as soon as possible.
- Migrate all content from side panes, header, and footer. Add images used before to new posts, re-use descriptions from old image database (TXT).
- Take elkement live and test redirects and preview images (social networks).
- If OK: Take the other apps live.
- Fix bugs
- Turn on redirects for old ASP pages.
- Watch results in web master tools.
- Inform Google about new URLs (Web Master Tools)
I've built the underlying 'flat-file database' (Details in this post), and my not yet public site has these features now:
- Menu bar from pages.
- Show all postings on home page
- Recent posts and archive in left bar.
- Tag cloud in right bar, tags created by grouping all posts' meta data.
- 'Tag page': Show all posts tagged with a specific tag.
- Indicate category of current posting by highlighting category in the menu.
- Highlight currently clicked article in archive.
- Menu page contains custom text plus automatically created list of all postings in this category.
- Automatic creation of RSS feed.
- CSS stylesheet and responsive design.
- 'Nice' URLs - ASP.NET Routing.
Currently I am painstakingly migrating snippets of content to new counterparts / articles / text files.
For testing I am using a layout similar to my Wordpress.com's blog design now: