INT 21h AH=3Ah removes the directory named by the ASCIIZ path at DS:DX: RD (RMDIR) at the prompt. It removes only an empty directory, one with no entries but "." and "..".

AX (CF = 1)Why
3path not found: there is no such directory
5access denied: not empty, read-only, or the root directory
16 (10h)it is the current directory of its drive

Hidden and system files count. DIR does not show them, so a directory can look empty and still fail with 5. To clear one, find its files with INT 21h AH=4Eh (with hidden and system bits in CX), clear a read-only attribute with INT 21h AH=43h, and delete each with INT 21h AH=41h; subdirectories have to be emptied and removed first.

No drive's current directory can be removed, not only the current drive's: change out of it first. The example goes into TESTDIR, tries to remove it from inside, then from its parent, then once more:

prints: rmdir while inside it: error 16 / rmdir from its parent: ok / rmdir once more: error 3
MOV AH, 39h ; make TESTDIR
MOV DX, OFFSET DIR1
INT 21h
MOV AH, 3Bh ; and go into it
MOV DX, OFFSET DIR1
INT 21h
MOV DX, OFFSET INM1
MOV AH, 09h
INT 21h
MOV AH, 3Ah ; DOS: remove the directory named at DS:DX
MOV DX, OFFSET SELF1 ; "..\TESTDIR": where we are
INT 21h
CALL RESULT ; error 16
MOV AH, 3Bh ; up to the parent
MOV DX, OFFSET UP1
INT 21h
MOV DX, OFFSET OUTM1
MOV AH, 09h
INT 21h
MOV AH, 3Ah
MOV DX, OFFSET DIR1
INT 21h
CALL RESULT ; ok
MOV DX, OFFSET AGAINM1
MOV AH, 09h
INT 21h
MOV AH, 3Ah
MOV DX, OFFSET DIR1
INT 21h
CALL RESULT ; error 3: it is gone
DIR1 DB "TESTDIR", 0
SELF1 DB "..\TESTDIR", 0
UP1 DB "..", 0
INM1 DB "rmdir while inside it: $"
OUTM1 DB "rmdir from its parent: $"
AGAINM1 DB "rmdir once more: $"
RESULT PROC NEAR ; print "ok", or "error" and AX when CF = 1
PUSHF ; keeps AX and CF for the caller
PUSH AX
MOV DX, OFFSET OKMSG
JNC RESULT1
MOV DX, OFFSET ERRMSG
MOV AH, 09h
INT 21h
POP AX
PUSH AX
CALL PDEC ; the error code
MOV DX, OFFSET CRLF
RESULT1:
MOV AH, 09h
INT 21h
POP AX
POPF
RET
RESULT ENDP
OKMSG DB "ok", 13, 10, "$"
ERRMSG DB "error $"
PDEC PROC NEAR ; print AX in decimal (changes AX BX CX DX)
MOV BX, 10
XOR CX, CX ; digits on the stack
PDEC1:
XOR DX, DX
DIV BX ; AX = AX / 10, DX = the last digit
PUSH DX
INC CX
OR AX, AX
JNZ PDEC1
PDEC2:
POP DX ; the digits come back first to last
ADD DL, "0"
MOV AH, 02h
INT 21h
LOOP PDEC2
RET
PDEC ENDP
CRLF DB 13, 10, "$"

DOSBox-X makes the error 16 check itself before it asks the host to delete the folder, since the host knows nothing about DOS's current directories. When the host refuses for another reason, DOSBox-X reports 5, as DOS does for a directory that is not empty.