One plus One equals Zero

(disclaimer: this is a rant. no real insight should be expected)

Came across this well known mathematical/philosophical phrase by chance (somethines: "one plus one equals one" or "one plus two equals zero").

And this is what happend:

We came up with this product that provide a kind of ETL pipeline over data using sparks. This pipeline give a simplified read-process-write for multiple data-sources using multiple processing scripts into any number of sink data targets.

For the initial POC release we based our work on an existing product, that is used to share data between organizations. This share product provided (being in its early version itself) only one share method, called "Copy-Based".

Copy based looks like that:

1. I share a "pointer" to my data with you
2. When you want it, you create a "snapshot"
3. You ask for a copy of this snapshot into your own storage.
4. Now you have a copy, and you can sell it.

This method give you an authentic copy of the data, and is simpler to secure on the backend. Disadvantages: It gives you stale data by definision, require storage, and is pretty slow.

A more advanced option is "In-Place" method, that goes a bit like this:

1. I "allow" you to read my data.
2. You access the shared "pointer" and read my data "directly"
3. You are smarter now.

The second method saves storage cost, limit trafic cost to when you need it, and ensure the data you read is always up to date. It is also faster inherently, because you access the data as soon as it is available. Disadvantages: It is pretty hard to secure on the backend, and lose the inherent snapshot capability of the Copy-Based method.

When we leveraged this product, copy based method fit our need perferctly for the early version, and this is how we used it:

1. The user give us the simplified pipeline.
2. We identify all the data it consumes, and copy it into our processing environment.
3. We process away like there is no tomorrow.
4. We copy the results out into the users' storage.

Then we said "Yes, we can do all that, but we are using Copy-Based method, and maybe some day, we'll do the In-Place fandango."

After a while we had some kind of architectual review with someone, and he looked at our product, and then he looked some more, and then he coughed and said:

"You are doing In-Place dude. As soon as the magik-pipeline is triggered, it pulls the data, process it, and spit out a result. This is what In-Place is for!"

We where shocked and bewildered. It meant that we will have to rename all our Enums! But it also meant something else. 

It was funny.

Our product provided In-Place processing that needs to create a snapshot from the source data, copy it for processing, need storage to save the result before you can access it, and by then it is already stale. Because it takes forever.

In other words, we managed to get all the disadvantages of Copy-Based and In-Place in one go!

Then I realized that in this case, One plus One equals Zero.

The Lazy Developer Paradox

Developers are Lazy. Yes. I said it, and for this reason alone, it is absolutely true. It may be insulting, or prejudice, but there is no doubt about it. Developers just hate working.

I am referring of course to software developers. Not Real-Estate or Photograph developers. They might be lazy too, only I wouldn't know anything about that. All I know is what I know (duh...). And I am a code developers for some time now.

A real developer will understand immediately what I mean. And she will not be surprised if I added that this same developer will code for days without food or sleep when the challenge is in front of her. And the root for this behavior, is one. No matter how'd you spin it, the main drive, or spark, or dark urge in the developer heart is that he simply DO NOT WANT TO DO THE WORK!

Now, if there is still a reader out there that do not agree (and I'm sure there is, because people are inherently evil), I'll lay out the example that supports my claim.

I used to work in a code factory (a.k.a: software-company) that made this one software product, and sold it all over the world. In those days, software used to come in expensive cardboard boxes, filled with many plastic diskettes numbered "one" to "lots", and a 20 lb user manual. All of these had to be introduced in the language of the country it was sold in. No french man would have payed good Franks to have his computer insulted by an English speaking User-Manual.

I worked in the "translation" department. Not that I know any language other then my mothers-tough (hell, I'm not even sure I have my mothers' tough). What I had to do is take out the English content out of the user manual, and the Software's menu and the Diagram captions, wrap it all in a neat little translatable package, send it to the translation-factory (or whatever these poor university graduate slaves were called), receive it back when they finished, and push it back into the Manual/Menu/Caption relevant areas.

I was bored out of my mind. It was repetitive, mind numbing, HARD work. When I could not stand it anymore, I wrote a small VB tool that just did it all for me. It opened the Adobe file editor, copied the text, converted it, merged it with the software menu, and repeat and rains. It was a silly piece of code, but it worked. It did the job a bit faster, and it allowed me not to do it.

Yes.
I was completely off the hook.

No more hard work.
All that was left to do is press a button....

And learn Visual Basic.
And Adobe-Publisher DDL.
And Trans-Script file format.
And Windows-Resource-File format.
And Debug, and Improve, and Fix-Bugs and Redesign and Refactor and on, and on, and on.
Forever....

So here is the Lazy Developer Paradox:

Developers (well, to be honest, at least some of them) write code to do their work for them. Especially when it is a simple task, with little-to-no mental challenge, and too much repetitive actions. And all these developers come across the same inescapable conclusion: Writing code that replace you, is VERY VERY HARD WORK.

And then they write code that write the code for you.

And this is how the next generation of programing language is born.

Helpful Errors

"Errors. The grate unknown. No single discipline may cover this weird and mysterious creature for all technologies...
...because what is error-handling after all? It is a curious man looking through a keyhole, the keyhole of mishap, trying to know what's going on." (Jacques Yves Cousteau)

I'm working on a new code base, and see this "errors.*" code file, And some of the errors (although, I'm happy to say, not most), reminds me an old joke:

A pilot needs to land a plane in Heathrow. It is about 4:00 AM, the fog is like a thick blanket, and his navigational equipment is a bit off. To put it in the simplest terms: the pilot is lost.
He flies around for some time, lower and lower, looking for some kind of landmark to help him find his bearings. Suddenly he notice a man looking at him out of a window. He slows down, roll down his own window, and call to the man:
"Excuse me, sir. Can you please tell me where I am?"
"Of course" - answer the man - "you are in a plane".
The pilot feels relief, turn 23.6 degrees left, continue for 13.51 miles, and land on runway 24B, just south of Heathrows' Terminal 5.
The control tower, that was aware of the pilots' predicament, asks him:
"How did you find your way?"
The man tells his story, and adds: "As soon as the man in the window answered, I knew I was over the Microsoft building. It is exactly their kind of help..."

So lets' imagine this scenario:
1. I sign-in into my e-commerce service.
2. I add some items to my basket.
3. I proceed to the checkout, and try to pay with my credit card.
4. The service return an error, saying: "CloseOrderValidationFailed".

What should I do? Should I pay again? Should I remove an Item? Should I join the navy?

CloseOrderValidationFailed is one of the errors in the "errors.*" file I looked at. It have a message, and status and code. The only thing it lacks, is meaning.

So what is a meaningful error? Here are some pointers:

• It should help the user. If the user see's it, and say "oh, right. my bad..." - it is excellent.
• It should never say "F**k you!". An error like "Payment Failed" is the same as saying "We do not serve people like you". Not only that it is not helpful, it is an insult.
• It should fit it's audience. You are in a bad place when your application display something like: "Error #26748. Please contact someone that know what this error mean!". Even if you know that the users that see this error cannot do anything about it, you must help him remain positive. And I do not mean "Please do not panic", but more like: "It appear we have encountered a problem (...appearance may be deceiving...). Please try to restart your application. If it do not help, try to restart your iPad. If you do not have an iPad, please contact us in the following address, and we will be happy to sell you one." - although it looks like a joke, most people will restart the app after the second sentence, and with some luck it will give you time to fix the problem on your end.

 Finally, some guidelines I constructed, mostly to help me when I need to design an error:

1. Talk about the "Why" rather then the "What": "Money withdrawal failed" is what happened. "Withdrawal amount exceeded the daily limit of 99$" is why it happened. Although it is helpful to know the former, I would rather be informed of the later.
2. Suggest a solution, even if it is an obvious one: "Cannot access the internet. Please make sure your LAN cable is connected properly" helped me personally more then once.
3. Be secured, but not frustrating: "Id check digit do not match RFC 3044" is definitely too much information, but "Make sure you omit leading zeros in the Id input" might actually save repeated head injury.
4. Be multi-dimensional: Error should be concise when displayed to the end user, and extensive when logged to the system audit. So "LoyaltyMembershipExpired" error will include an informative message saying "Your 'Pockimon-Go' membership expired. Please go to your town center to renew it", and also some other properties like "source", "account-id" and "stack-trace" that will be visible to the technician on the back-end.
5. Things get lost in translation: Any useful information that may be given by the error should be kept as close to the error source as possible. Do not fret to have your code, message, all your message translations and your formatting variables in the same error class (or near it).It is better not to relay on a client to convert your code into something readable if you can help it.
6. Be unique, but do not quench other information sources: Allow your errors to be unique in the system you manage, but always allow to include errors from external sources. There are cases when a third party system will articulate why it failed much better. Make it part of your error design.

And the most impotent thing: Never underestimate the need for good error design. From the error-class to the error-auditing - all this is an important foundation to a system that can last beyond the proof-of-concept phase.

Big Data Architecture - Comic strip

A very good architect help with the impotent questions:

How to implement a requirement (a.k.a. "The eMail Chain")

When you work in a big company like ours, with a big budget and a lot of participants, you tend to learn a lot about how product development should be done.

One of the interesting aspect of modern R&D delivery processes is the use of e-mail to help mystify the business requirements, justify the existence of the different positions and pass the time.

Because some people might find this general statement a bit too general, and even, god forbid, silly, I will give an example.

Listed below is an actual e-mail chain* started by one of the architects in our company. I only changed 3 things:

1. The order - a real chain (like a blog) starts at the end.
2. I added an interpretation of what the writer actually meant (in [slant square brackets]) for people that do not read co-operative.
3. The names - like someone actually care.

And now, here is the trick:

Although it can be fun to read the whole thing through, you may read ONLY the slanted text to understand my point.

GL
* email-chain: electronic mail correspondence between more then 2 parties.
 

---

From: Tectus, Archie
Sent: Wednesday, January 20, 2016 16:51
To: Leader, Tim; Loper, Dave
Cc: Yorass, Witkis
Subject: Extractor item name/description in Complexi-T

Tim Dave,

I do not see the code that fills the item name/description in CT.

Any reason for that? Or I missed something in the code.

This is mandatory field for all the systems that need to read this common crap-pile.

Archie

(..copy paste of some code...)

[Someone said there is a problem in the system that according to my title I should understand, but know nothing about. 

Please take the blame for me not doing my job.

Also, according to my title, I have access to the code, so do not mess with me!]

_____________________________________________
From: Loper, Dave
Sent: Wednesday, January 20, 2016 9:52 AM
To: Tectus, Archie
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

Please send a reference to the original requirement that was submitted to development

[I will not take the blame for your screw-ups. I know you are full of shit.
Also: I really don't care.]

_____________________________________________
From: Tectus, Archie
Sent: Wednesday, January 20, 2016 16:54
To: Loper, Dave
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

? what requirement relevant here?

I am asking you if you parse the name of the item in CT parser…

I can’t find it in the code (appear only in Intrica-CT parser).

So the crap-pile present the list of items w/o description or name.

Could you please let me know if this exist in the code? Because I can’t find it.

Archi

[I do not do requirements. I expect things to just be there. 
I have no idea how to actually participate in a development cycle, and I think I am too important to be part of one.
Now, take the blame, or I will do what my grandfather did!]


_____________________________________________

From: Loper, Dave
Sent: Wednesday, January 20, 2016 9:56 AM
To: Tectus, Archie
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

No you did not.

You said:

This is mandatory field for all the systems that need to read this common crap-pile.

It implies a requirement.

Just send the req. number, so I can track down the test that created the code you are looking for.

Dave

[1. I know you have no idea of what you are talking about. 
 2. This is an e-mail chain, and I know how to scroll down. 
 3. I really, REALLY, do not care what your Grandfather did (or anything else for that mater)]

_____________________________________________

From: Tectus, Archie
Sent: Wednesday, January 20, 2016 17:00
To: Loper, Dave
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

Don’t know which req. relevant for that.

I see that in the crap-pile, there is field for ItemName.

But I can’t find it in the parser code.

Could you please help me to find this specific code that parse into this specific field?

Archie

[I got it!
You want to find the requirement that you did not fulfill, so Witkis can have a concrete proof you are to blame for my incompetence.
Grate idea! Godspeed.]

_____________________________________________

From: Loper, Dave
Sent: Wednesday, January 20, 2016 10:00 AM
To: Tectus, Archie
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

If it was not required it will not be there.

[Stick it in your rosy backside!]


_____________________________________________

From: Tectus, Archie
Sent: Wednesday, January 20, 2016 10:31 AM
To: Loper, Dave; Nihilist, B. Zane
Cc: Leader, Tim; Yorass, Witkis; Toaner, Prodah
Subject: RE: Extractor item name/description in Complexi-T

Zane,

I see that the item name appear in the common crap-pile. Is this a requirement? Are there any req. number that relevant to that?

Archie

[Demit! I have just realized I can find at least one more employee to take the blame for me! 
I just love monstrous corporations!!!]


_____________________________________________

From: Nihilist, B. Zane
Sent: Thursday, January 21, 2016 3:36 AM
To: Loper, Dave; Tectus, Archie; Toaner, Prodah
Cc: Leader, Tim; Yorass, Witkis; 
Subject: RE: Extractor item name/description in Complexi-T

Hi Archie,

We are investigating this issue.

Can you please explain what is the requirement for the product name.

Thanks,

Billy.

[Got your email.
We will ignore it as soon as possible.
Also: Business requirements is not something we know about here, in the BA department. So you are most welcome to solve this problem by yourself.]

_____________________________________________

From: Tectus, Archie
Sent: Thursday, January 21, 2016 14:05
To: Loper, Dave; Nihilist, B. Zane; Toaner, Prodah
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

1.     It’s already exist in the crap-pile contract – but only the ICT parser provide this information ,other parsers are not, and this is not acceptable.
2.      It was defined in the past for mandatory and also defined in the consumer format as important.
3.      Based on the 90 days project that we are doing now, there are requirements for several systems that consume this crap, without calling other services (like car-wash). This data exists in the original pile and there are no reason to use other services.

Archie

[1. I will not agree to this kind of insolence.
 2. I never did my job, and not going to start now.
 3. I know for a fact something I have 5 months to fabricate any way I like.]


_____________________________________________

From: Toaner, Prodah
Sent: Thursday, January 21, 2016 7:49 AM
To: Loper, Dave; Nihilist, B. Zane; Tectus, Archie
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

Hi Archie,

The product name does not exist in the Complexi-T pile, not in the Least nor the Last version.

It will require additional effort from Last to add this to their pile.

We are investigating this, will update you soon. 

Thanks,
Prodah

[As I already perceived, you know nothing.
I will continue ignoring you when I'm less busy]

_____________________________________________

From: Tectus, Archie
Sent: Thursday, January 21, 2016 15:00
To: Toaner, Prodah; Loper, Dave; Nihilist, B. Zane; Manger, Smalltim; Nagger, Persistent
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

Yes I see.

On the other hand I see that other systems (like un-expected-name and ICT) are sending this information.

I am curious how the consumer system is working? Seems that it appears in the system as it is in the original pile….but we are not providing the original pile…so what is missing here?

Also about another consumer? How is it presented today the items drill down?

Archi

[I do not need to know. I am above this pay grade.
But I added some other people in the "To" field, to make sure everybody know I am working my ass off!]

_____________________________________________
From: Manger, Smalltim
Sent: Thursday, January 21, 2016 15:35
To: Tectus, Archie; Toaner, Prodah; Loper, Dave; Nihilist, B. Zane; Nagger, Persistent
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

Inside XXX they are generating the HTML irelevant-page based off of the crap-pile.

System and Other-System are doing direct calls to XXX to get the page.

Kroocky made this decision to have this logic here.  I have talked with Chakie about this being an issue longer term when different formats (font, etc) will be required for customers.

Smalltim 

[Whaff. Whaff. Whaff. Wiggle my tail. Whaff.]

_____________________________________________
From: Tectus, Archie
Sent: Thursday, January 21, 2016 16:58
To: Toaner, Prodah; Loper, Dave; Nihilist, B. Zane; Manger, Smalltim; Nagger, Persistent
Cc: Leader, Tim; Yorass, Witkis
Subject: RE: Extractor item name/description in Complexi-T

Proda,

Right now we are in a situation where some fields in the common crap-pile are missing for CT customers.(like, product name)

That means that in case we are going to use generic reports, analytics ,etc , central systems that are based on the common crap-pile will failed to work.

Could you please come with final list of all the properties that are mandatory in the common crap-pile.

Archie

[Right now we are in a situation we should not have been in if a proper architecture was provided to the relevant product.
So I strongly ask the business team to provide a set of requirements I already failed to deliver in the past, so I may fail to do it again.
And please be quick about it. My time is extremely valuable!!!]

---

Goodness... that was long, wasn't it?

If you manged to get this far, apart of being entitled to several diligence awards, I can try and give you my little piece of mind.

Email chains are the hallmark of inefficiency and organizational-dysfunction. It expose to the fullest the inability of people to work together when central guidance is expected. 

When you see something like this in your work place, you know for sure one thing: you have a twat for a manager (represented in the example by "Witkis Yorass"). 

A competent manager would stop this chain at the 4th link, channeling it into some kind of productive meeting or work-item. 

In this specific example, the participant of the conversation became so useless, due to low moral and/or skill-set, that the manager in charge just gave up, and let the ball roll unhindered.

When you see something like that in your work place, it is a good excuse to go out and find another work place. Or a rich widow. Or a sugar-daddy. Or win the lottery. Yes. This is it. See you in my Malibu house.

Some thoughts about BDD

Bob asked me about BDD.
"Behavioral Driven Design?" said I.
"Well... something like that... maybe..." he announced knowingly.
"I met it once" I stuttered.
"Think about it!" said our leader, and vanished in a puff of smoke.

So I thought about it.

The first problem I have with "Behavior Driven Development" is the name itself. Every time someone say it, I get confused. Its' a paradox. Like Military Intelligence.
Behavior is something you observe. A software behavior is driven by the way it was developed. Not the other way around. When I hear "BDD" I think that I need to develop something to behave in some way that will drive me to develop something. Then I must breath into a paper bag.

Of course, what was actually MEANT is that someone will say how he wants the system to behave, and then the developer will do it. Which is actually what software development is all about, and I'm pretty sure it worked that way before 2006.

So what is the difference? The Givenwhenthen.

The Givenwhenthen is a magical creature. It have the head of a business-spec with the body of a unit-test, and the legs of a DSL. When you look at some of the frameworks developed to run it, be it a wiki with a "Test" button, or one of the cucumis sativus family, you get the feeling that what was given when Dan North spoke is much bigger then the product it drives.

Don't get me wrong, I'm all for providing automated tests that describe the business requirements. I'm a big believer in TDD and ATDD. I also really like detailed business requirements. I just think that maintaining a specialized client just to write unit/integration tests is a nuisance more then a benefit.

A business analyst think of requirements in terms of acceptance details. In his mind he sees client application the end-user operates, and devise the best way this app will provide the business needs. He think about reports, and what kind of information should be there. He creates elaborated excel to verify the system actually works. He listen to clients request and decipher what they actually need.

What a BA do not do is write code. And Givenwhenthen is code. Not because it must follow a syntax to fit a parser (which it does), and not because it does not work without writing code that gives it meaning (like most requirements) - it is because only a developer can write, modify and maintain it.

I was there. In the BDD zone. Writing structured semi-formal artifacts to represent behavior, and then writing adapters to pass it a quasi-business-model, that we hooked to the actual system, that was so far from the test at that point that you could never understand why it was red. At some point we just gave up. The effort was so big, we decided to spend it on developing the product itself. And the BA people never even looked at it. So we went back to TDD, and ATDD and integration tests.

So yes. We are doing BDD now. Only it looks just like a unit test, and we call it integration. And when it is done right, it represent the expected behavior, and give a good hint of the requirements... and here is my last reservation.

BDD worries me because it emphasize behavior over requirements. And behavior changes from one client to another, even if it actually uses the same set of services. The FAEMA Enova allow dosed delivery programming, while the ZjNac F305 supply 7 premixed drinks. But both uses the same water measurement service. Because both need to handle the water volume for different behaviors.

If BA will start writing unit-tests, we'll be in a worse place then we where when Dan North presented the TDD challenge. The BA will have to ask themselves all these TDD questions, and might miss out on the Developer to Business-Expert dialog completely.

And then we can kiss DDD goodbye.

Highly anticipated unexpected announcement

Our development group leader scheduled a meeting on a short notice. The summoned participants included what is considered in our department the "best-of-the-best", strongest, fastest and most-likely-to-suckseed. Oh yes, and me.

I am a code-developer.
At least it is implied from my linked-in profile.
What it actually say is: "Team Leader in Coffeex", which is not true.

For one, I am not a team leader. I am the worst leader I ever met. The last time I lead a team it was made out of 2 other people, that resigned 2 weeks after the team was created. I suspect it was that, or suicide, so at least it means there is an alternative to me.

Second fact is that "Coffeex" do not exist anymore. It was bought in a multimillion transaction about a year ago. How many millions (you may ask)? Lots. More then ten, less then a million. Who bought us (you may ask {you are a bit of a nuisance})? Non other then the well known, mighty, multi-everything cooperative: "NCM United". Who the hell is that (you may ask again - stretching my patience very thin)? Google it!

"NCM United", originally "National Coffee Machines", is an American based coffee-machines, hot-beverages-automation and brewing-software company. Its' main strength is in the hardware area, but it wanted to expend it's software market, especially these days that you can pour fresh cappuccino strait out of your iPhone 8.

Coffeex developed one of the most advanced coffee making software in the world (or at least - the most complicated one). NCM wrote an 8 zeros check, and now it had the most advanced coffee making software etc.
In the 15 month that passed since the deal, NCM did everything it can to kill the software it paid so much to acquire. Moving customers from the Coffeex solution to outdated vending machines, reducing the product range to a minimum, and limiting the new project bids to... well, zero.

And after all this, we arrive to the present day, when we were all sitting around, doing pretty much nothing, waiting for the final closure announcement, that this meeting happened.

Bob (that is our group leader) entered the meeting room, hung a "do not disturb" sign on the door, and locked it. After sweeping for bugs, he made his statement:

"Gentleman.
"I am here to tell you about the most recent decision made by the all mighty NCM management on the other side of the world.
"After this weekend the Coffeex software is no more. You few chosen ones will replace it with a brand new software. A better software. A faster software. A software better suited for today's xanthine alkaloid consumption needs.
"You have 48 hours to decide if you want to be part of this vague and shapeless endeavor, or stay behind working on the crumbling remains of the old, stale and insignificant working product.
"This project, should you decide to accept it, is top secret, and should not be mentioned by any of you for the next 120 minutes.
"This message will self-destruct in five seconds."

. . .

This was the introduction to what is going to be (probably) the most mundane and expected NCM project, known now only by its' code name: NEMO.
My next post will give the details of this initial meeting, that I cannot disclose at the moment, for it is only an hour and a half since it began.