JavaScript vs. Python for preprocessors and data conversion

[tepples]

For roughly the past seven years that I've been doing NESdev, I have written data conversion and source code preprocessing tools in Python because I can get them working faster than if I had written them in C++. These tools include PNG to CHR, PNG+palette to CHR+NAM, Pently's MML-like language to ca65 assembly, source code shuffling to add buffer overflow canaries and make prototype leaks traceable, and NESASM assembly to ca65 assembly for Action 53. But I try to keep my options open.

(I myself prefer JavaScript as scripting language of choice, but that is just my opinion and you can use Python if you prefer to.)

I'm curious about the advantages of JavaScript over Python. Pip vs. NPM looks like a wash as far as I can tell, but I have a couple concerns.

One is more practical. I imagine that more people have Python installed than Node, though Windows comes with neither. It does come with Windows Script Host (cscript.exe), which interprets a JavaScript dialect called JScript, but the standard libraries bundled with Node and WSH are incompatible in several ways. For example, reading files in Node looks nothing like reading files in JScript. So any command-line program to be used by a wide audience would have to wrap both Node's I/O and WSH's I/O.

Crockford rant ahead

Another is whether Douglas Crockford, author of the JSON spec and JavaScript: The Good Parts, cautions against using bitwise operations in JavaScript, which if followed would limit its use for data conversion tools. The &, |, <<, and >>> operators are mentioned only in the chapter "The Bad Parts":

In Java, the bitwise operators work with integers. JavaScript doesn't have integers. It only has double precision floating-point numbers. So, the bitwise operators convert their number operands into integers, do their business, and then convert them back. In most languages, these operators are very close to the hardware and very fast. In JavaScript, they are very far from the hardware and very slow. JavaScript is rarely used for doing bit manipulation.

As a result, in JavaScript programs, it is more likely that & is a mistyped && operator. The presence of the bitwise operators reduces some of the language's redundancy, making it easier for bugs to hide.

JSLint, Mr. Crockford's static analysis tool to enforce some conventions described in The Good Parts, has a "tolerate bitwise operators" checkbox that defaults off. In its manual, he defends this similarly: "The bitwise operators are rarely used in JavaScript programs, so it is usually more likely that & is a mistyping of && than that it indicates a bitwise and."

But then Mr. Crockford is known for being opinionated. JSLint has no options to control indentation style other than "tolerate whitespace mess". In particular, its requirement of 4-space indentation is incompatible with the 2-space indentation to which Firefox Scratchpad defaults. (You can't change the indent level from within Scratchpad; the setting is more hidden. You have to open the developer tools on a web page, click the gear, and scroll down to "Editor Preferences". The page about Scratchpad doesn't even mention it; it mentions only the ability to enable Emacs or Vim key bindings. But perhaps this ire should be on Bugzilla, not here.) It used to have an indent level option, but Mr. Crockford removed it in this change that diff thinks is a complete rewrite, and he closes feature requests to reintroduce it within hours without comment. And he puts a vague and non-free field of use restriction "The Software shall be used for Good, not Evil." in the modified MIT license used for his JSON reference implementation and for JSLint.

Is it wisest A. to ignore some of Mr. Crockford's axes to grind, B. to use a language other than JavaScript for data conversion tools, or C. to exercise a third option that you are about to describe?

[zzo38]

PNG to CHR (but currently not PNG+palette to CHR+NAM) is something that can be done using a combination of programs I had written in C (you may combine by shell scripts if you want to).

About bitwise operations, what I can say is I think V8 (the JavaScript engine used in Node.js) will optimize it anyways (I don't quite know the details of this though). I don't like Crockford's opinions myself.

However, if someone writes in Python it is still possible that someone else will write a version in JavaScript too, and vice versa.

Let's see what everyone else on here says, too.

[dougeff]

You worry too much. Write your tools in whatever language best suits you.

Python source is not 100% portable, due to syntax differences between version 2 & 3. But, it's much better than compiled C++, which isn't portable at all...and few people will want to compile it.

I don't have a strong opinion of NODE, so I won't comment.

[adam_smasher]

Python and JS are at roughly the same level of abstraction; they're close enough that you could probably write a source-to-source translator without too much sweat. If you're comfortable with Python I don't think you'll gain much by switching.

Unless you're churning absurd amounts of data, the higher cost of bitwise operators in JS isn't going to be a problem (and even then, you'll likely be IO bound in either case). And Crockford's advice is outdated anyway; JS The Good Parts was published in 2008, long before V8 pushed browser developers to speed their JS engines up to near native speeds. If your work load *does* call for it, speed is actually a very compelling reason to use JS over Python.

[tepples]

Python source is not 100% portable, due to syntax differences between version 2 & 3.

The same is true of Perl 5 and 6, or VB 6 and VB.NET. But future statements and pip install six help developers make polyglot Python 2/3 programs. And since Python 3.3, it's possible for 2 and 3 to coexist on a PC. And I've ported all my new projects to Python 3 since I started work on 240p Test Suite late last year. After The Curse of Possum Hollow goes gold, I'll be putting more stuff on GitHub, and all the Python in those will be Python 3. Later I plan to open a new topic about my post-Curse plans.

Python and JS are at roughly the same level of abstraction; they're close enough that you could probably write a source-to-source translator without too much sweat.

Some previous attempts to transpile Python to JavaScript failed, but today I learned that Transcrypt exists.

And you have a good point about The Good Parts predating V8, and in fact predating my serious attempts at NESdev.

[Banshaku]

My opinion is:

- If you write tools for yourself then use the tools you are comfortable with
- If you plan to share your tools the choice should still be yours, you don't have to follow the latest fad
- If you write tools as a team then decide whith tools everyone can maintain and use
! By a team I means people you work with everyday, not a community you participate in

There is no point to convert code to another platform unless you stop using the previous one and the cost of porting the code is lower then maintaining the old one.

Don't reinvent the wheel unless you have a good reason to do so. See your time invested on writing tools as a cost. This should be enough to give you an idea if the cost of doing so is worth it instead of working on your main project.

[tokumaru]

I personally prefer PHP for writing scripts and tools. It's simple enough to setup as a portable application (just put in a folder and change a config file to enable the extensions you need) and I'm comfortable with the syntax from having used it a lot at work.

[Rahsennor]

I write all my tools in C, because that's what I'm comfortable with, and it can be compiled on every major operating system. My tools are pretty simple though.

Speaking of, I wrote a BMP to CHR+NAM+palette tool, if anyone wants it. Was thinking of combining it with a simple ROM to make a drag-and-drop nametable preview generator for artists, if I ever find the time.

[tepples]

I write all my tools in C, because that's what I'm comfortable with, and it can be compiled on every major operating system.

On a Mac, it needs 4.7 GB Xcode, which can pose a problem unless you either live in wired broadband's service area or carry your Mac into a coffee shop.

Speaking of, I wrote a BMP to CHR+NAM+palette tool, if anyone wants it.

I wonder how you guess the palette.

[calima]

Xcode is included on the OS X dvd, so no internet strawman for that case.

[pstalcup]

Personally, I've written my dev tools in ruby (I am mostly looking at GB right now, though, if that makes a difference).

I use Ruby at my job, so the biggest reason I've selected it is familiarity. In the past, I've used Python extensively for this sort of pipelining, because I find it to be quick to use.

[tepples]

Xcode is included on the OS X dvd, so no internet strawman for that case.

How many Macs include a DVD reader anymore?

[Banshaku]

@Tepples

If you want to compile C code for terminal it should be possible without xCode by installing the command line tools for your OS version (download from Apple site with free developer account). If you really need xCode then in the case you cannot download it yourself I think it doesn't hurt to ask a friend for such thing once in a while. Maybe the Apple Store will let you download if you ask them. (I know, you don't have a Mac).

And if you are serious Mac developer you have xCode anyway since it's the only way to develop apps easily in swift, objective-c etc.

The point is yes, C code is portable in general, no need to nitpick about edge case.

@calima

New version of OSX doesn't come with a DVD anymore so you have to download everything from Apple. It was still possible to get it from the dvd until Snow Leopard I think. With today internet it maybe less an issue than 8 years ago. Got xCode 8 yesterday and waiting for MacOS Sierra next week.

[zzo38]

I personally prefer PHP for writing scripts and tools. It's simple enough to setup as a portable application (just put in a folder and change a config file to enable the extensions you need) and I'm comfortable with the syntax from having used it a lot at work.

Yes, this is an advantage of PHP, although in my opinion PHP isn't a very good programming language so now I use JavaScript (although I do have PHP installed, to run programs I have previously written in PHP (such as FurryScript and a OpenID provider) and to run a program I found elsewhere to insert files into a ZIP archive from stdin which is also written in PHP).

On a Mac, it needs 4.7 GB Xcode...

I think you can install GCC with the Homebrew package manager too now. Objective-C is included, but Swift isn't.

Speaking of, I wrote a BMP to CHR+NAM+palette tool, if anyone wants it.

Do you have available source code as FOSS? It may be useful to some people if so.

The point is yes, C code is portable in general, no need to nitpick about edge case.

There are a few things to consider if porting to Windows, such as some POSIX functions are not available (although you might not need them), and you can use #ifdef _WIN32 to conditionally compile code to switch stdin/stdout to binary mode if you are using stdin/stdout for binary. Existing programs could probably be changed in this way without too much difficulty if you are compiling a C program for Windows. Porting between Linux and Macintosh is probably easier.

[mikejmoffitt]

Both homebrew and Macports require the XCode Command Line Tools to be installed, at a minimum. However, this is a bad argument against using C, as Banshaku has pointed out. You don't need to serve every possible edge case like "Mac owners who want to get into NesDEV using my tools but have never done development on the computer in question in a rural setting". They are your tools, so make them for your use case. Or shall we consider that some users may consider phpBB a violation of their freedoms, and will not wish to use the website? A line has to be drawn for how far you're willing to go for all users.