Thinking about algorithms

[Rhea]

I titled it that was because in the Lounge there's a "posts like this" algorithm at the bottom and the posts are years old. So hey why not.

Algorithms sometimes do what they claim, but other times they can give wrong info, or they can stagnate. So how do we write them to be more durable?

 

[rousseau]

Algorithmic design is an iterative process that's constrained by a few things. To name a few:

- business resources (money)
- IT talent
- political will
- business imperatives
- deadlines

So usually in the software world the main focus is getting things 'out the door' under budget, before your competitors, and without killing anybody. This means that software and algorithms are going to be inherently imperfect. If we had unlimited know-how, time, money.. no problem, but that's not how it works. So in practice what we see are companies getting working products out the door, and then refining them over time toward a chosen end-goal.

 

[Rhea]

I would have expect algorithm design (for enhanced experience and/or extras - those used to get to know the user and predict next desire) to be somewhat of a separate (parallel) path to software design. I think of it as a “module” that would be used across several (many) platforms, and so I would have thought there was a higher level of sophistication for it thatn for the user interface itself.

Maybe they always make it new?

 

[rousseau]

I would have expect algorithm design (for enhanced experience and/or extras - those used to get to know the user and predict next desire) to be somewhat of a separate (parallel) path to software design. I think of it as a “module” that would be used across several (many) platforms, and so I would have thought there was a higher level of sophistication for it thatn for the user interface itself.

Maybe they always make it new?

True, but still a part of the overall system design with the same constraints.

You can have a bunch of incredibly talented engineers with PhDs, who think they've designed the world's best system. They release it to the wild and it fucks up in new and unexpected ways that they never considered (think Facebook).. things happen things happen.. new design.

 

[Phil Scott]

I titled it that was because in the Lounge there's a "posts like this" algorithm at the bottom and the posts are years old. So hey why not.

Algorithms sometimes do what they claim, but other times they can give wrong info, or they can stagnate. So how do we write them to be more durable?

Formally prove them correct!

This is a long solved problem, even if few programmers know what formal proof is. A less solved problem is how to let an algorithm evolve with its proof-of-correctness as it is adapted to changing requirements.

- - - Updated - - -

You can have a bunch of incredibly talented engineers with PhDs, who think they've designed the world's best system. They release it to the wild and it fucks up in new and unexpected ways that they never considered (think Facebook).. things happen things happen.. new design.

I don't think Facebook was built by PhDs. I thought it was built by a PHP script kiddie.

 

[rousseau]

I titled it that was because in the Lounge there's a "posts like this" algorithm at the bottom and the posts are years old. So hey why not.

Algorithms sometimes do what they claim, but other times they can give wrong info, or they can stagnate. So how do we write them to be more durable?

Formally prove them correct!

This is a long solved problem, even if few programmers know what formal proof is. A less solved problem is how to let an algorithm evolve with its proof-of-correctness as it is adapted to changing requirements.

- - - Updated - - -

You can have a bunch of incredibly talented engineers with PhDs, who think they've designed the world's best system. They release it to the wild and it fucks up in new and unexpected ways that they never considered (think Facebook).. things happen things happen.. new design.

I don't think Facebook was built by PhDs. I thought it was built by a PHP script kiddie.

Well Zuckerberg was a programming savant during the initial build, but these days I can't begin to imagine the depth of their systems.

I just use them as a prime example of a company that has high standards for engineers, and you'd think would be able to make perfect decisions, but who very noticeably hasn't. Hence when you're building software, or software modules, or anything, you make a bunch of assumptions first, many of which turn out to be untrue or not what you expected.

 

[Phil Scott]

Formally prove them correct!

This is a long solved problem, even if few programmers know what formal proof is. A less solved problem is how to let an algorithm evolve with its proof-of-correctness as it is adapted to changing requirements.

- - - Updated - - -

I don't think Facebook was built by PhDs. I thought it was built by a PHP script kiddie.

Well Zuckerberg was a programming savant during the initial build, but these days I can't begin to imagine the depth of their systems.

I just use them as a prime example of a company that has high standards for engineers, and you'd think would be able to make perfect decisions, but who very noticeably hasn't. Hence when you're building software, or software modules, or anything, you make a bunch of assumptions first, many of which turn out to be untrue or not what you expected.

I don't know much about Zuck. What made him a programming savant?

If you want to write robust code, the first thing you do is avoid PHP like the plague. Zuck completely failed on that.

 

[rousseau]

Formally prove them correct!

This is a long solved problem, even if few programmers know what formal proof is. A less solved problem is how to let an algorithm evolve with its proof-of-correctness as it is adapted to changing requirements.

- - - Updated - - -

I don't think Facebook was built by PhDs. I thought it was built by a PHP script kiddie.

Well Zuckerberg was a programming savant during the initial build, but these days I can't begin to imagine the depth of their systems.

I just use them as a prime example of a company that has high standards for engineers, and you'd think would be able to make perfect decisions, but who very noticeably hasn't. Hence when you're building software, or software modules, or anything, you make a bunch of assumptions first, many of which turn out to be untrue or not what you expected.

I don't know much about Zuck. What made him a programming savant?

I forget where I heard it now, but IIRC his parents hired a private programming tutor for him when he was a young kid. Later on he was studying comp sci at Harvard. I think right around the time he was building Facebook he was getting big offers, as people at the top of our field usually do.

If you want to write robust code, the first thing you do is avoid PHP like the plague. Zuck completely failed on that.

Well the guy's a multi-billionaire now so I can forgive his language of choice

 

[Phil Scott]

I forget where I heard it now, but IIRC his parents hired a private programming tutor for him when he was a young kid.

He was struggling so badly at school that he had to get private tuition ?

It doesn't mean much. Lots of us kids of the 80s grew up learning to code.

Well the guy's a multi-billionaire now so I can forgive his language of choice

The lesson here is that becoming a billionaire doesn't require being able to write robust code, which is the point made in the OP: almost no-one writes code with formal correctness in mind. Why not? Well, here's the easy answer: you are more likely to become a billionaire by shipping code as quickly as you can even if it barely works than by making it robust.

I gave up on the idea that the money follows merit in software in the late 90s. I was very late to that view compared to others.

 

[rousseau]

I forget where I heard it now, but IIRC his parents hired a private programming tutor for him when he was a young kid.

He was struggling so badly at school that he had to get private tuition ?

It doesn't mean much. Lots of us kids of the 80s grew up learning to code.

Well the guy's a multi-billionaire now so I can forgive his language of choice

The lesson here is that becoming a billionaire doesn't require being able to write robust code, which is the point made in the OP: almost no-one writes code with formal correctness in mind. Why not? Well, here's the easy answer: you are more likely to become a billionaire by shipping code as quickly as you can even if it barely works than by making it robust.

I gave up on the idea that the money follows merit in software in the late 90s. I was very late to that view compared to others.

Raises the obvious question of: 'what qualifies as merit'

- The guy trying to write the most technically beautiful piece of software ever made
- The guy with a billion users on his platform

The technical side of software doesn't exist in a vacuum, it's only there to support a business. So the primary goal is supporting the business, the secondary goal is technical correctness.

This is where the constraints I mentioned come in. Like anything else, being 'moral, good, socially just, accurate' comes second to business imperatives, political will, and energy/time constraints. Add to this the context of coding for a business where you're forced to support the profitable rather than correct thing, and you get a serious problem if you actually want socially sustainable software.

 

[rousseau]

He was struggling so badly at school that he had to get private tuition ?

It doesn't mean much. Lots of us kids of the 80s grew up learning to code.

The lesson here is that becoming a billionaire doesn't require being able to write robust code, which is the point made in the OP: almost no-one writes code with formal correctness in mind. Why not? Well, here's the easy answer: you are more likely to become a billionaire by shipping code as quickly as you can even if it barely works than by making it robust.

I gave up on the idea that the money follows merit in software in the late 90s. I was very late to that view compared to others.

Raises the obvious question of: 'what qualifies as merit'

- The guy trying to write the most technically beautiful piece of software ever made
- The guy with a billion users on his platform

The technical side of software doesn't exist in a vacuum, it's only there to support a business. So the primary goal is supporting the business, the secondary goal is technical correctness.

This is where the constraints I mentioned come in. Like anything else, being 'moral, good, socially just, accurate' comes second to business imperatives, political will, and energy/time constraints. Add to this the context of coding for a business where you're forced to support the profitable rather than correct thing, and you get a serious problem if you actually want socially sustainable software.

If there's a punch-line or answer here, I think it might actually be 'convincing people who control budgets and specs to actually understand what they're doing'. Because it's great in theory to 'get things out the door', but lots of companies shoot themselves in the foot just as much as they become successful.

 

[Rhea]

If there's a punch-line or answer here, I think it might actually be 'convincing people who control budgets and specs to actually understand what they're doing'. Because it's great in theory to 'get things out the door', but lots of companies shoot themselves in the foot just as much as they become successful.

That's true in manufacturing as well as coding...

 

[Phil Scott]

The technical side of software doesn't exist in a vacuum, it's only there to support a business.

Not all software is business software. Indeed, virtually none of the software I use is business software. And I'll take Linux over Windows any day of the week.

The open social media protocols still being drafted through by the W3C aren't being cooked up to support a business, and I'm more interested in those than I am in Facebook.

 

[Phil Scott]

It's also worth pointing out that, historically, software companies don't get sued when their software spectacularly fails. So far, there has been no litigation over the ridiculous mistakes of Heartbleed and Cloudbleed, which were both mistakes that are only possible because of a poor choice of programming language.

We don't hold real engineers to such lax standards. If a bridge collapses whilst you are driving over it, you don't shrug and say "well, that's business needs for you."

The great trick that software companies played on the public was to declare that all software has bugs, and that when crappy software fails, that's to be expected. Heaven forbid programmers be expected to write code that's solid. Again, this is why "software engineer" is a term that gets nothing but derision from real engineers.

 

[rousseau]

The technical side of software doesn't exist in a vacuum, it's only there to support a business.

Not all software is business software. Indeed, virtually none of the software I use is business software. And I'll take Linux over Windows any day of the week.

The open social media protocols still being drafted through by the W3C aren't being cooked up to support a business, and I'm more interested in those than I am in Facebook.

Well sure, not all software exists to support some kind of organization, but the point was that the usual brand of development that people are doing to make a living is to support some type of org. In that scenario your goal isn't to write good code, it's to drive profit and decrease costs. The problem is that this is often at odds with writing ideal code, which I guess is the crux of the problem mentioned in the OP.

If we're talking Open Source that's a different thing altogether, but only in the sense that it's under a different set of constraints.

Basic point is that bad software is an artefact of it's constraints. There's idealism, and then there's reality.

- - - Updated - - -

It's also worth pointing out that, historically, software companies don't get sued when their software spectacularly fails. So far, there has been no litigation over the ridiculous mistakes of Heartbleed and Cloudbleed, which were both mistakes that are only possible because of a poor choice of programming language.

We don't hold real engineers to such lax standards. If a bridge collapses whilst you are driving over it, you don't shrug and say "well, that's business needs for you."

The great trick that software companies played on the public was to declare that all software has bugs, and that when crappy software fails, that's to be expected. Heaven forbid programmers be expected to write code that's solid. Again, this is why "software engineer" is a term that gets nothing but derision from real engineers.

I agree. To me this is primarily because software is such a new and poorly understood field, even by the people who take part in it.

I'd think that it's only a matter of time before higher standards become the norm, but as of now that seems to be like trying to throw a dart at a high-speed train.

 

[lpetrich]

I titled it that was because in the Lounge there's a "posts like this" algorithm at the bottom and the posts are years old. So hey why not.

Algorithms sometimes do what they claim, but other times they can give wrong info, or they can stagnate. So how do we write them to be more durable?

There are algorithms and there are algorithms. Lots of different ones for doing lots of different things.

Some of them are exact, like integer arithmetic and data-arrangement actions. Like Euclid's algorithm for finding a greatest common divisor of some numbers -- one does not need to factor them. Or sorting algorithms.

Some of them are approximate, though with well-defined intended results. Like numerical integration, finding the area under a function curve.

Some of them do not have well-defined intended results, like the similar-posts algorithm. Ideally, one would want the forum software to understand the contents of the posts and then use that understanding to find which posts are similar. However, we have yet to create any AI software that can do such understanding. So we use such measures as similarity in word content.

Rhea, that may explain the odd results that you have been seeing -- the forum software using some simplistic comparison algorithm.

 

[Rhea]

And of course there is a dollar cost to that.

Interesting to think about well-defined intended results (implying of course that unintended results are always possible) and looselyly defined ones...

 

[steve_bank]

Have you considered that there are algorithms thinking about you?

 

[lpetrich]

Have you considered that there are algorithms thinking about you?

What do you have in mind?

 

[Phil Scott]

Have you considered that there are algorithms thinking about you?

What do you have in mind?

Automated greeting cards, I'll bet.