INT 21h AH=5Bh is INT 21h AH=3Ch with a safety catch. If no file of that name exists, it creates one and returns a handle open for reading and writing, exactly as 3Ch does. If one exists, nothing is touched and the call fails with CF = 1 and AX = 50h, "file exists". The function came with DOS 3.0; DOS 2 does not have it.

Why not test first with INT 21h AH=3Dh and create when the open fails? Because between the test and the create another program (a TSR, or another machine on a network) can make the file. 5Bh checks and creates in one step, and that makes it the tool for lock files: whoever creates LOCK.TMP first owns the resource, everyone after gets 50h until the owner deletes it.

prints two lines: "first 5Bh: created" and "second 5Bh: error 50"
MOV     AH, 5Bh       ; DOS: create TEST.TXT, which must be new
XOR     CX, CX
MOV     DX, OFFSET FNAME
INT     21h
CALL    CHECK
MOV     BX, AX
MOV     AH, 3Eh       ; close it
INT     21h
MOV     DX, OFFSET M1
MOV     AH, 09h       ; DOS: print the string at DS:DX
INT     21h
MOV     DX, OFFSET M2
MOV     AH, 09h       ; DOS: print the string at DS:DX
INT     21h
MOV     AH, 5Bh       ; again: now the file exists
XOR     CX, CX
MOV     DX, OFFSET FNAME
INT     21h
CALL    REPORT        ; CF = 1, AX = 50h
MOV     AH, 41h       ; delete TEST.TXT
MOV     DX, OFFSET FNAME
INT     21h
 
FNAME   DB      "TEST.TXT", 0
M1      DB      "first 5Bh: created", 13, 10, "$"
M2      DB      "second 5Bh: $"
CRLF    DB      13, 10, "$"
CHECK   PROC    NEAR  ; CF = 1: print the error code in AL and exit
JNC     CHECK1
PUSH    AX
MOV     DX, OFFSET FAILMSG
MOV     AH, 09h
INT     21h
POP     AX
CALL    PHEXB
MOV     AX, 4C01h     ; exit with code 1
INT     21h
CHECK1:
RET
CHECK   ENDP
FAILMSG DB      "failed: error $"
REPORT  PROC    NEAR  ; CF = 0: "ok"; CF = 1: "error" and AL
JC      REPORT1
MOV     DX, OFFSET OKMSG
MOV     AH, 09h
INT     21h
RET
REPORT1:
PUSH    AX
MOV     DX, OFFSET ERRMSG
MOV     AH, 09h
INT     21h
POP     AX
CALL    PHEXB         ; the error code
MOV     DX, OFFSET CRLF
MOV     AH, 09h
INT     21h
RET
REPORT  ENDP
OKMSG   DB      "ok", 13, 10, "$"
ERRMSG  DB      "error $"
PHEXB   PROC    NEAR  ; print AL as two hex digits
PUSH    CX
PUSH    DX
MOV     DH, AL        ; keep the byte
MOV     CL, 4
SHR     AL, CL        ; the high digit first
MOV     CX, 2
PHEXB1:
AND     AL, 0Fh
ADD     AL, "0"
CMP     AL, "9"
JBE     PHEXB2
ADD     AL, 7         ; 10..15 -> "A".."F"
PHEXB2:
MOV     DL, AL
MOV     AH, 02h       ; DOS: print DL
INT     21h
MOV     AL, DH        ; then the low digit
LOOP    PHEXB1
POP     DX
POP     CX
RET
PHEXB   ENDP

The other errors are those of 3Ch: 03h for a directory in the path that does not exist, 04h when no handle is free, 05h when the directory cannot take another entry. 50h is the only new one, and it is the one a lock-file loop waits on.

Relatives

DOS 3 also added 5Ah, which creates a file with a unique name in a given directory and writes the chosen name back into your buffer: the way to make a temporary file. DOS 4 added 6Ch, extended open / create, which does the work of 3Ch, 3Dh and 5Bh in one call, with flags that say what to do when the file exists and when it does not.

DOSBox-X implements 5Bh as an open to test, then a create. A DOS program gets the same answers; only a host program changing the same folder at the same instant could slip in between.