C
Installing and Using Visual Studio

The Microsoft Macro Assembler (MASM), Microsoft C++ compiler, Microsoft linker, and other tools this book uses are all available in the Microsoft Visual Studio package. At the time of this writing, you can download the Visual Studio Community edition for Windows at https://visualstudio.microsoft.com/vs/community/. Of course, URLs change over time. A web-based search for Microsoft Visual Studio download should lead you to the appropriate spot.

C.1 Installing Visual Studio Community

Once you download the Visual Studio Community edition, run the installer program. This appendix does not provide step-by-step directions as Microsoft is famous for completely changing the user interface of programs even when minor updates occur. Any directions appearing here would probably be obsolete when you try to run them. However, the main thing you want to do is ensure that you download and install the Microsoft Visual C++ desktop tools.

C.2 Creating a Command Line Prompt for MASM

To use the Microsoft Visual C++ (MSVC) compiler and MASM, we need to initialize the environment by using a batch file provided by Visual Studio and then leave the command line interpreter (CLI) open so we can build and run programs. We have two options: use an environment created by the Visual Studio installer, or create a custom environment.

At the time of this writing, the Visual Studio 2019 installer creates various CLI environments:

Developer Command Prompt for VS 2019
Developer PowerShell for VS 2019
x64 Native Tools Command Prompt for VS 2019
x64_x86 Cross Tools Command Prompt for VS 2019
x86 Native Tools Command Prompt for VS 2019
x86_x64 Cross Tools Command Prompt for VS 2019

You can find these by clicking Start (the Windows icon) on the Windows taskbar and then navigating to and clicking the Visual Studio 2019 folder. x86 refers to 32-bit, and x64 refers to 64-bit versions of Windows.

The Developer Command Prompt, Developer PowerShell, x86 Native Tools, and x64_x86 Cross Tools target the 32-bit versions of Windows, so they are outside the scope of this book. x86_x64 Cross Tools targets 64-bit Windows, but the tools available in the environment are themselves 32-bit. Basically, these are the tools for people running a 32-bit version of Windows. x64 Native Tools are for people targeting and running a 64-bit version of Windows. The 32-bit versions of Windows are rare today, so we have not used nor tested this book’s code under x86_x64 Cross Tools. In theory, it should work to assemble and compile 64-bit code, but we would not be able to run it in this 32-bit environment.

x64 Native Tools running under 64-bit Windows is what we have used and tested. If you right-click x64 Native Tools, you can pin it to Start, or if you select More, you can pin it to the taskbar.

Alternatively, you can create your own custom environment, and we will now go through that process. We’ll create a shortcut to a MASM-ready command line prompt by using the following steps:

Find the batch file named vcvars64.bat (or something similar). If you cannot find vcvars64.bat, try vcvarsall.bat instead. At the time of writing this chapter (using Visual Studio 2019), I found the vcvars64.bat file in the following directory: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\.

Note

vcvars.bat or vcvars32.bat will not work (these set up the environment variables for the 32-bit version of the assembler and C++ compiler, which we don’t want to use).
Create a shortcut to the file (by right-clicking it in the Windows Explorer and selecting Create Shortcut from the pop-up). Move this shortcut to your Windows desktop and rename it VSCmdLine.
Right-click the shortcut icon on the desktop and click Properties▶Shortcut. Find the Target text box that contains the path to the vcvars64.bat file; for example:
```
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
```
Add the prefix cmd /k in front of this path:
```
cmd /k "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
```
The cmd command is the Microsoft cmd.exe command line interpreter. The /k option tells cmd.exe to execute the command that follows (that is, the vcvars64.bat file) and then leave the window open when the command finishes execution. Now, when you double-click the shortcut icon on the desktop, it will initialize all the environment variables and leave the command window open so you can execute the Visual Studio tools (for example, MASM and MSVC) from the command line.

If you can’t find vcvars64.bat but there is a vcvarsall.bat, also add x64 to the end of the command line:
```
cmd /k "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
```
Before closing the shortcut’s Properties dialog, modify the Start In text box to contain C:\ or another directory where you will normally be working when first starting the Visual Studio command line tools.
Double-click the shortcut icon on the desktop; you should be presented with a command window that has text like the following:
```
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.9.0
** Copyright (c) 2019 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
```
From the command line, type ml64. This should produce output similar to the following:
```
C:\>ml64
Microsoft (R) Macro Assembler (x64) Version 14.28.29910.0
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: ML64 [options] filelist [/link linkoptions]
Run "ML64 /help" or "ML64 /?" for more info
```
Although MASM is complaining that you haven’t supplied a filename to compile, the fact that you’ve gotten this message means that ml64.exe is in the execution path, so the system has properly set up the environment variables so you can run the Microsoft Macro Assembler.

As a final test, execute the cl command to verify that you can run MSVC. You should get output similar to the following:

C:\>cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29910 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [option...] filename... [/link linkoption...]

Finally, as one last check, locate the Visual Studio application in the Windows Start menu. Click it and verify that this brings up the Visual Studio IDE. If you like, you can make a copy of this shortcut and place it on the desktop so you can bring up Visual Studio by double-clicking the shortcut icon.

C.3 Editing, Assembling, and Running a MASM Source File

You will use a text editor of some sort to create and maintain MASM assembly language source files. If you’re not already familiar with Visual Studio and want an environment that’s easier to learn and use, consider downloading the (free) Notepad++ text editor application. Notepad++ provides excellent support for MASM, is fast, and is easy to learn and use. Regardless of which text editor you choose (I use a commercial product called CodeWright), the first step is to create a simple assembly language source file.

MASM requires that all source files have a .asm suffix, so create the file hw64.asm with your editor and enter the following text into that file:

includelib kernel32.lib

        extrn __imp_GetStdHandle:proc
        extrn __imp_WriteFile:proc

        .CODE
hwStr   byte    "Hello World!"
hwLen   =       $-hwStr

main    PROC
        
; On entry, stack is aligned at 8 mod 16. Setting aside 8
; bytes for "bytesWritten" ensures that calls in main have
; their stack aligned to 16 bytes (8 mod 16 inside function).
        
        lea     rbx, hwStr
        sub     rsp, 8
        mov     rdi, rsp        ; Hold # of bytes written here

; Note: must set aside 32 bytes (20h) for shadow registers for
; parameters (just do this once for all functions). 
; Also, WriteFile has a 5th argument (which is NULL), 
; so we must set aside 8 bytes to hold that pointer (and
; initialize it to zero). Finally, the stack must always be 
; 16-byte-aligned, so reserve another 8 bytes of storage
; to ensure this.

; Shadow storage for args (always 30h bytes).

        sub     rsp, 030h 
                
; Handle = GetStdHandle(-11);
; Single argument passed in ECX.
; Handle returned in RAX.

        mov     rcx, -11        ; STD_OUTPUT
        call    qword ptr __imp_GetStdHandle 
                
; WriteFile(handle, "Hello World!", 12, &bytesWritten, NULL);
; Zero out (set to NULL) "LPOverlapped" argument:
        
        mov     qword ptr [rsp + 4 * 8], 0  ; 5th argument on stack
        
        mov     r9, rdi         ; Address of "bytesWritten" in R9
        mov     r8d, hwLen      ; Length of string to write in R8D 
        lea     rdx, hwStr      ; Ptr to string data in RDX
        mov     rcx, rax        ; File handle passed in RCX
        call    qword ptr __imp_WriteFile
        add     rsp, 38h
        ret
main    ENDP
        END

This (pure) assembly language program is offered without explanation. Various chapters in this book explain the machine instructions.

Look back at the source code, and you’ll see the first line is as follows:

includelib kernel32.lib

The kernel32.lib is a Windows library that includes, among other things, the GetStdHandle and WriteFile functions this assembly language program uses. The Visual Studio installation includes this file and, presumably, the vcvars64.bat file will put it in an include path so the linker can find it. If you have problems assembling and linking the program (in the next step), simply make a copy of this file (wherever you can find it in the Visual Studio installation) and include that copy in the directory where you are building the hw64.asm file.

To compile (assemble) this file, open the command window (whose shortcut you created earlier) to get a command prompt. Then enter the following command:

ml64 hw64.asm /link /subsystem:console /entry:main

Assuming you entered the code without error, the command window should have output similar to the following:

C:\MASM64>ml64 hw64.asm /link /subsystem:console /entry:main
Microsoft (R) Macro Assembler (x64) Version 14.28.29910.0
Copyright (C) Microsoft Corporation.  All rights reserved.

 Assembling: hw64.asm
Microsoft (R) Incremental Linker Version 14.28.29910.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/OUT:hw64.exe
hw64.obj
/subsystem:console
/entry:main

You can run the hw64.exe output file that this assembly produces by typing the command hw64 at the command line prompt. The output should be the following:

C:\MASM64>hw64
Hello World!