Lee Lindley Scratchpad

Creating Multiple Row Headers with ExcelGen PL/SQL Spreadsheet Generator

Sun, 27 Nov 2022 00:00:00 -0500

Introduction

Marc Bleron’s ExcelGen package version 3 has added multi-table sheets, merged cells, and writing to individual cells. This allows us to construct workbooks with sheets that have multiple-row column headers. The business users I support are happy about that!

This article walks through an example of constructing a spreadsheet with a banner title, multiple-row column headers, automatic filtering, plus frozen rows and columns. We also set font and background colors in a fairly tame way, but it demonstrates the tools we have to make an attractive end product.

See the manual page at the above link to ExcelGen on github to get an idea of how this all works. Refer back to it as we walk through the example if you have questions.

Walking Through an Example

Here is an image of the spreadsheet we will produce.

Figure 1 - Multi-Row Spreadsheet Headers

The ‘+’ characters and NULLs in the first header row are to demonstrate a peculiarity of ExcelGen. You will notice in the image above that even though we told the tool to make the columns in our mini-table have a light gray background, the cells that did not have any content, did not get the background style applied. I opened an issue and expect it will be addressed in the next release.

The declared variables can be thought of as object handles to various parts of our logical spreadsheet. The values are returned by functions we call to create/declare those parts. In practice they are indexes into package global collections, but we do not need to know that.

DECLARE
    v_ctxId             ExcelGen.ctxHandle;
    v_sheetId           ExcelGen.sheetHandle;
    v_tableId           ExcelGen.tableHandle;
    v_headerTableId     ExcelGen.tableHandle;
    v_headerCellStyle   ExcelGen.cellStyleHandle;
    v_cellStyle         ExcelGen.cellStyleHandle;
    -- discussed below. There are other ways to accomplish this
    v_arr_headers       arr_arr_varchar2_udt := app_csv_pkg.split_clob_to_fields(
q'{+,,+,
First,Last,Department,Yearly
Name,Name,Name,Salary}'
                                                );

We want to use a small font for the report and for the report header to have bold text on a light gray background. To get started we create our workbook and two style handles we can assign as needed. You can use CSS syntax for styles, but I’m sticking with the familiar style construction from ExcelGen version 2 for now.

BEGIN
    -- create a workbook and some initial styles to use in assignments
    v_ctxId := ExcelGen.createContext;

    -- style to use for header cells
    v_headerCellStyle := ExcelGen.makeCellStyle(
        p_ctxId => v_ctxId
        ,p_font => ExcelGen.makeFont('Arial',8,p_b => TRUE)
        ,p_fill => ExcelGen.makePatternFill('solid','lightgray')
    );
    -- style to use inside the report proper
    v_cellStyle := ExcelGen.makeCellStyle(
        p_ctxId => v_ctxId
        ,p_font => ExcelGen.makeFont('Arial',8)
    );

Next we will create our first (and only) workbook sheet with the sheet name of ‘Employees’. We’ll give the tab a color just because we can (I don’t normally). Then we put a text value into the A1 cell of the sheet. This will be our report title banner at the top of the sheet.

We create another style for this cell on the fly that will automatically apply to the cells we merge together with A1 in a moment.

    -- add a sheet to the workbook
    v_sheetId := ExcelGen.addSheet(
        p_ctxId             => v_ctxId
        ,p_sheetName        => 'Employees'
        ,p_tabColor         => 'lime' 
    );

   -- place banner text in top left corner cell
    ExcelGen.putStringCell(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_rowIdx           => 1
        ,p_colIdx           => 1
        ,p_value            => 'Employee Salary Report'
        ,p_style            => ExcelGen.makeCellStyle(
                                    p_ctxId => v_ctxId
                                    ,p_font => ExcelGen.makeFont('Calibri',15,p_b => TRUE)
                                    ,p_fill => ExcelGen.makePatternFill('solid','lime')
                                    ,p_alignment => ExcelGen.makeAlignment(
                                        p_horizontal    => 'center'
                                        ,p_vertical     => 'center'
                                    )
                                )
    );

So far that is just in a single cell. We want to make it span across the top two rows of the report, centered. We could span a single row of cells and set the height of that row, but will merge across two rows here instead. We could get fancy and figure out the range from a number of rows and perhaps the number of columns in our header collection, but the water is muddy enough for now.

    ExcelGen.mergeCells(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_range            => 'A1:D2'
    );

Next, we want to create the first two rows of our column headers. We do not put our last row of the column headers here as they must be handled separately. We could use putStringCell again to do this, but I have a plan for extending the functionality of multi-row column header handling we’ll get to when we refactor. Therefor, we will create a cursor that returns our column headers as data (but only the first 2 rows of header, not the last), and add that as a table to our worksheet.

The task of generating a cursor from generic two dimensional collection object turned out to be harder than I expected. I covered it in Create a PL/SQL Cursor from a Nested Table of Nested Tables. If you are not ready to install package app_csv_pkg from my github repository, that article shows other ways you can do it.

You will notice that our anchor is one based. It can be confusing that absolute anchors are one based while relative anchors are zero based. I do not think I will be the last person to stumble and make mistakes with this.

We assign a style to the columns of our little 2 row table as well. Notice that we are using a style we call a header style for the data of this table. The table contains data we want to appear as column headers.

    v_headerTableId := ExcelGen.addTable(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_rc               => app_csv_pkg.get_cursor_from_collections(
                                    p_arr_arr       => v_arr_headers
                                    ,p_trim_rows    => 1 -- do not include last row of headers
                               )
        ,p_anchorColOffset => 1  -- we start in column 1
        ,p_anchorRowOffset => 3  -- leave 2 rows of room for our banner. we start in row 3
    );
    FOR i IN 1..4
    LOOP
        ExcelGen.setTableColumnProperties(
            p_ctxId         => v_ctxId
            , p_sheetId     => v_sheetId
            , p_tableId     => v_headerTableId
            , p_columnId    => i
            , p_style       => v_headerCellStyle
        );
    END LOOP;

Next, we add our real query to the sheet as a second table beneath the two row header table. We use relative positioning for the location of this table. The concepts of the anchor and relative positioning are covered well in the documentation. There is a nice picture with examples. I have gone back to it frequently when building this and other multi-table sheets and/or positioning of putCell calls. The zero based vs one based anchors and thinking in terms of whether you are operating at the sheet level or the table level can be a challenge.

    v_tableId := ExcelGen.addTable(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId 
        ,p_query            =>
q'{SELECT first_name, last_name, department_name , salary
FROM hr.employees e
INNER JOIN hr.departments d
    ON d.department_id = e.department_id
        AND department_name IN ('Finance', 'Executive')
ORDER BY salary DESC}'
        ,p_anchorTableId    => v_headerTableId
        ,p_anchorPosition   => ExcelGen.BOTTOM_LEFT
        ,p_anchorColOffset  => 0
        ,p_anchorRowOffset  => 1
    );

For our data table we want to set the header that goes with the data. We could have simply put the third header row in the table we constructed above, but if we did that we could not turn on autoFilter. It seems best to tell Excel that this is a header even if you do not need autoFilter.

We then override the column header values from what was in the query. You could have made the query column aliases do what you needed here and only had to set the column style properties, but this demonstrates how the query column aliases can be ignored.

Let me point out that setTableColumnProperties allows us to set the value of the column header, but the style we are applying is to this column in the data rows, not the header. The header style is determined in the call to setTableHeader.

The mixing of concepts between column header and column data in this function is unfortunate, but probably not worth making separate methods. Technically, it all applies to a table column and the distinction in styles between header and data is a subdivision of table column properties.

    ExcelGen.setTableHeader(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_tableId          => v_tableId
        ,p_style            => v_headerCellStyle
        ,p_autoFilter       => TRUE
    );
    FOR i in 1..4
    LOOP
        ExcelGen.setTableColumnProperties(
            p_ctxId         => v_ctxId
            ,p_sheetId      => v_sheetId
            ,p_tableId      => v_tableId
            ,p_columnId     => i
            ,p_style        => v_cellStyle
            -- override the column header from query column aliases
            ,p_columnName   => v_arr_headers(3)(i)
        );
    END LOOP;

We need to set the number format for the fourth data column, Salary. We could have created a style that included the number format and selectively used it with setTableColumnProperties above.

So far I’ve been choosing to set the date, timestamp and number formats separately from the the styles.

Since setColumnFormat operates at the sheet level it applies to the header rows too, but since those values are strings and not numbers, it doesn’t matter.

    -- This is at the sheet level
    ExcelGen.setColumnFormat(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_columnId         => 4
        ,p_format           => '#,##0.00_);(#,##0.00)' -- an accounting number style
    );

Next we want to freeze our column headers and the first two columns of data (the first and last name values). In this example freezing the first two columns is not very useful since the total number of columns is so small, but it is instructive to see how it is done.

Specifying the top left corner of the region that is allowed to scroll is what Excel does too, but we typically think in terms of which columns and rows are frozen. Too bad. We need to specify where the scrollable active pane starts.

The row we make top of the scrollable region is the first one with data – two rows for the banner, three rows of header, so row six is first row of data. Column 3 is the first one we want in the scrollable pane.

    -- freeze row headers and first two columns (aka setting active pane)
    ExcelGen.setSheetProperties(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_activePaneAnchorRef => ExcelGen.makeCellRef(
            p_colIdx            => 3
            ,p_rowIdx           => 6
        )
    );

We are done declaring the content of the sheet. We finish by executing everything to create the workbook file and clean up after ourselves with the closeContext.

    --  ExcelGen.createFile(ctx, 'TEST_DIR', 'multi-row-header.xlsx');
    ExcelGen.createFile(v_ctxId, 'TMP_DIR', 'multi-row-header.xlsx');
    ExcelGen.closeContext(v_ctxId);
  
END;
/

You saw the output spreadsheet at the top of the article.

Conclusion

This journey through creating a spreadsheet touches on both the nuts and bolts of general spreadsheet creation with ExcelGen and also now some bells and whistles with a banner and multi-row column headers.

It is a lot of code though. I intend to refactor into a reusable utility procedure for multi-row column headers. I was going to cover that utility in another article. I wound up doing the work for an employer on their dime, so it isn’t mine to share. If you have read this far, I’m sure you could create your own given the pattern shown here.

Appendix - Full Code Listing

DECLARE
    v_ctxId             ExcelGen.ctxHandle;
    v_sheetId           ExcelGen.sheetHandle;
    v_tableId           ExcelGen.tableHandle;
    v_headerTableId     ExcelGen.tableHandle;
    v_headerCellStyle   ExcelGen.cellStyleHandle;
    v_cellStyle         ExcelGen.cellStyleHandle;
    v_arr_headers       arr_arr_varchar2_udt := app_csv_pkg.split_clob_to_fields(
q'{+,,+,
First,Last,Department,Yearly
Name,Name,Name,Salary}'
                                                );
BEGIN
    -- create a workbook and some initial styles to use in assignments
    v_ctxId := ExcelGen.createContext;
    v_headerCellStyle := ExcelGen.makeCellStyle(
        p_ctxId => v_ctxId
        ,p_font => ExcelGen.makeFont('Arial',8,p_b => TRUE)
        ,p_fill => ExcelGen.makePatternFill('solid','lightgray')
    );
    v_cellStyle := ExcelGen.makeCellStyle(
        p_ctxId => v_ctxId
        ,p_font => ExcelGen.makeFont('Arial',8)
    );

    -- add a sheet to the workbook
    v_sheetId := ExcelGen.addSheet(
        p_ctxId             => v_ctxId
        ,p_sheetName        => 'Employees'
        ,p_tabColor         => 'lime' 
    );
    -- place banner text in top left corner cell
    ExcelGen.putStringCell(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_rowIdx           => 1
        ,p_colIdx           => 1
        ,p_value            => 'Employee Salary Report'
        ,p_style            => ExcelGen.makeCellStyle(
                                    p_ctxId => v_ctxId
                                    ,p_font => ExcelGen.makeFont('Calibri',15,p_b => TRUE)
                                    ,p_fill => ExcelGen.makePatternFill('solid','lime')
                                    ,p_alignment => ExcelGen.makeAlignment(
                                        p_horizontal    => 'center'
                                        ,p_vertical     => 'center'
                                    )
                                )
    );
    ExcelGen.mergeCells(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_range            => 'A1:D2'
    );


    v_headerTableId := ExcelGen.addTable(
        p_ctxId             => v_ctxId
        ,p_sheetId          =>  v_sheetId
        ,p_rc               => app_csv_pkg.get_cursor_from_collections(
                                    p_arr_arr       => v_arr_headers
                                    ,p_trim_rows    => 1 -- do not include last row of headers
                               )
        ,p_anchorColOffset => 1
        ,p_anchorRowOffset => 3  -- leave room for our banner
    );
    FOR i IN 1..4
    LOOP
        ExcelGen.setTableColumnProperties(
            p_ctxId         => v_ctxId
            , p_sheetId     => v_sheetId
            , p_tableId     => v_headerTableId
            , p_columnId    => i
            , p_style       => v_headerCellStyle
        );
    END LOOP;

    v_tableId := ExcelGen.addTable(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId 
        ,p_query            =>
q'{SELECT first_name, last_name, department_name , salary
FROM hr.employees e
INNER JOIN hr.departments d
    ON d.department_id = e.department_id
        AND department_name IN ('Finance', 'Executive')
ORDER BY salary DESC}'
        ,p_anchorTableId    => v_headerTableId
        ,p_anchorPosition   => ExcelGen.BOTTOM_LEFT
        ,p_anchorColOffset  => 0
        ,p_anchorRowOffset  => 1
    );

    ExcelGen.setTableHeader(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_tableId          => v_tableId
        ,p_style            => v_headerCellStyle
        ,p_autoFilter       => TRUE
    );
    FOR i in 1..4
    LOOP
        ExcelGen.setTableColumnProperties(
            p_ctxId         => v_ctxId
            ,p_sheetId      => v_sheetId
            ,p_tableId      => v_tableId
            ,p_columnId     => i
            ,p_style        => v_cellStyle
            -- override the column header from query column aliases
            ,p_columnName   => v_arr_headers(3)(i)
        );
    END LOOP;

    -- This is at the sheet level
    ExcelGen.setColumnFormat(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_columnId         => 4
        ,p_format           => '#,##0.00_);(#,##0.00)' -- an accounting number style
    );

    -- freeze row headers and first two columns (aka setting active pane)
    ExcelGen.setSheetProperties(
        p_ctxId             => v_ctxId
        ,p_sheetId          => v_sheetId
        ,p_activePaneAnchorRef => ExcelGen.makeCellRef(
            p_colIdx            => 3
            ,p_rowIdx           => 6
        )
    );

    --  ExcelGen.createFile(ctx, 'TEST_DIR', 'multi-row-header.xlsx');
    ExcelGen.createFile(v_ctxId, 'TMP_DIR', 'multi-row-header.xlsx');
    ExcelGen.closeContext(v_ctxId);
  
END;
/

Create a PL/SQL Cursor from a Nested Table of Nested Tables

Thu, 24 Nov 2022 22:00:00 -0500

Introduction

How can we store a generic two dimensional table in PL/SQL, bind it to SQL as a data source and generate a cursor from it? We have a variable number of columns, so a Record Type is not practical. We’ll explore some options.

A Well Defined Problem

We have a table with known number of columns and column names. We want to create a structure that represents spreadsheet column headers for a report that reads from this table. I better repeat that. We are not discussing a structure for holding the data from the table. We want a similar structure to hold multiple rows of column header information for a spreadsheet.

Here is an image of the spreadsheet we will produce.

Figure 1 - Multi-Row Spreadsheet Headers Try 1

Rows 3, 4 and 5 are our column header rows. The columns are from the hr.employees and hr.departments tables. The tool we use to generate the spreadsheet requires that we provide a SYS_REFCURSOR or a SQL query string as an input parameter. We’ll work with a SYS_REFCURSOR here to include bind variables.

We could generate a SQL query string as:

    OPEN v_sys_refcursor FOR
q'{SELECT '+', NULL, '+', NULL FROM dual
UNION ALL
SELECT 'First', 'Last', 'Department', 'Yearly' FROM dual
UNION ALL
SELECT 'Name', 'Name', 'Name', 'Salary' FROM dual}'
    ;

I’m less than impressed. We could create a pair of public types in our package (declared in the package specification so that the SQL engine can see it):

    TYPE t_header_rec IS RECORD(
        first_name VARCHAR2(32)
        ,last_name VARCHAR2(32)
        ,department_name VARCHAR2(32)
        ,yearly_salary VARCHAR2(32)
    );
    TYPE t_arr_header_rec IS TABLE OF t_header_rec;

Then we can assign the data to it like so in our procedure declaration:

    v_arr_headers   t_arr_header_rec := t_arr_header_rec(
        t_header_rec('+',NULL,'+',NULL)
        ,t_header_rec('First','Last','Department','Yearly')
        ,t_header_rec('Name','Name','Name','Salary')
    );

And open a cursor that reads from it:

    OPEN v_sysrefcursor FOR q'{SELECT
        first_name, last_name, department_name, yearly_salary
FROM TABLE(:header_table)}' 
        USING v_arr_headers
    ;

There is no ORDER BY here. We are depending on the implementation to provide the rows in the same order as the collection. It seems a pretty safe bet, but I have searched long and hard for documentation that makes a promise on this. Oracle pulled the rug out from under us once before when we were dependent on the implementation of GROUP BY via a sort. Tom Kyte wrote adamantly that the only way you can guarantee the order of records is to use an ORDER BY clause. Yet I cannot think of a reason why the current implementation providing the records in the same order as the collection would change.

A Generic Solution

I want to refactor my larger problem of providing a data structure containing the header row content for any spreadsheet, not just this one report. We will not have a Record Type with a fixed number of columns. For each row we need a nested table of strings. We’ll define a schema level collection Object:

CREATE OR REPLACE TYPE arr_varchar2_udt AS TABLE OF VARCHAR2(4000);
/

You probably already have one of those available to you in your own schema or perhaps a common schema. Oracle ships with several of them.

We also need a nested table of those puppies to build our two dimensional structure in our PL/SQL program:

CREATE OR REPLACE TYPE arr_arr_varchar2_udt AS TABLE OF arr_varchar2_udt;
/

Now our procedure declaration becomes:

    v_arr_headers   arr_arr_varchar2_udt := arr_arr_varchar2_udt(
        arr_varchar2_udt('+',NULL,'+',NULL)
        ,arr_varchar2_udt('First','Last','Department','Yearly')
        ,arr_varchar2_udt('Name','Name','Name','Salary')
    );

There is a glitch in the matrix though. How do we extract the individual elements of each row in the SQL statement for the cursor? Here is one way.

Note the careful use of table aliases. Object access in SQL has a quirk where an alias for the table or Common Table Expression (CTE) name is required.

One triksie part here is the join in the CTE named b. We are joining to a column in our rowset from CTE named a. Think that through. For each row from a, we treat the column in that row like it is a table. When we join to it we are essentially unpivoting that column into multiple rows, one for each value in the array named col. CTE c has a separate record for every column on every line of our headers. We rank them in the order they came out of the collection so we can pivot them in the final select.

    -- v_sql declared as a CLOB
    v_sql := q'{WITH a AS (
        SELECT rownum AS rn, t.COLUMN_VALUE AS rec -- rec is a nested table arr_varchar2_udt
        FROM TABLE(:header_table) t
    )
    , b AS (
        SELECT rn, rownum AS crn, c.COLUMN_VALUE AS col
        FROM a t
        INNER JOIN TABLE(t.rec) c
            ON 1=1
    )
    , c AS ( -- could I have done math with rn and crn instead? not easily.
        SELECT rn, ROW_NUMBER() OVER (PARTITION BY rn ORDER BY crn) AS cn, col
        FROM b
    ) SELECT }';

    -- create a pivot aggregate for each column
    DECLARE
        l_comma VARCHAR2(1);
    BEGIN
        FOR i IN 1..v_arr_headers(1).COUNT
        LOOP
            v_sql := v_sql||l_comma||q'{
                MAX(CASE WHEN cn = }'||TO_CHAR(i)||' THEN col END) AS c'||TO_CHAR(i);
            l_comma := ',';
        END LOOP;
    END;

    v_sql := v_sql||q'{
        FROM c
        GROUP BY rn
        ORDER BY rn}';

    OPEN v_sysrefcursor FOR v_sql USING v_arr_headers;

Once again we are depending on Oracle to provide the rows in the same order as the collection. I’m hoping I’m retired before that changes. Yeah, yeah, you think I’m worried about a boogeyman that will never pop out from under the bed. Told ya, I’ve seen it before.

With some formatting to make it easier to see the output, here is what I ran in sqlplus for a test:

set pagesize 0
set trimspool on
set linesize 100
column c1 format a24
column c2 format a24
column c3 format a24
column c4 format a24
var curs REFCURSOR
DECLARE
    v_arr_headers   arr_arr_varchar2_udt := arr_arr_varchar2_udt(
        arr_varchar2_udt('+',NULL,'+',NULL)
        ,arr_varchar2_udt('First','Last','Department','Yearly')
        ,arr_varchar2_udt('Name','Name','Name','Salary')
    );
    v_sql CLOB;
    --v_sysrefcursor SYS_REFCURSOR;
BEGIN
    v_sql := q'{WITH a AS (
        SELECT rownum AS rn, t.COLUMN_VALUE AS rec -- rec is a nested table arr_varchar2_udt
        FROM TABLE(:header_table) t
    )
    , b AS (
        SELECT rn, rownum AS crn, c.COLUMN_VALUE AS col
        FROM a t
        INNER JOIN TABLE(t.rec) c
            ON 1=1
    )
    , c AS ( -- could I have done math with rn and crn instead? not easily.
        SELECT rn, ROW_NUMBER() OVER (PARTITION BY rn ORDER BY crn) AS cn, col
        FROM b
    ) SELECT }';

    -- create a pivot aggregate for each column
    DECLARE
        l_comma VARCHAR2(1);
    BEGIN
        FOR i IN 1..v_arr_headers(1).COUNT
        LOOP
            v_sql := v_sql||l_comma||q'{
                MAX(CASE WHEN cn = }'||TO_CHAR(i)||' THEN col END) AS c'||TO_CHAR(i);
            l_comma := ',';
        END LOOP;
    END;

    v_sql := v_sql||q'{
        FROM c
        GROUP BY rn
        ORDER BY rn}';

    OPEN :curs FOR v_sql USING v_arr_headers;
END;
/
print curs

And the output:

SQL> @x.sql
PL/SQL procedure successfully completed.

+                                                 +
First                    Last                     Department               Yearly
Name                     Name                     Name                     Salary

SQL>

It works. It’s ugly. Can we make it better?

Refactor Using a WITH Clause PL/SQL Function

In PL/SQL we can get the value of a collection element with the syntax v_arr(i) where i is the index value. There is no equivalent syntax in SQL. That is why we had to do the self-join in the example above.

Rather than self-joining to our collection of columns to break them into separate rows, then pivoting them back into columns afterwards, let’s give ourselves a Function that we can call in SQL to do it. We could define the function in a package or standalone, but as of Oracle 12.1 we can do it in-line with our SQL.

I like this much better than the convolutions above:

set pagesize 0
set trimspool on
set linesize 100
column c1 format a24
column c2 format a24
column c3 format a24
column c4 format a24
var curs REFCURSOR
DECLARE
    v_arr_headers   arr_arr_varchar2_udt := arr_arr_varchar2_udt(
        arr_varchar2_udt('+',NULL,'+',NULL)
        ,arr_varchar2_udt('First','Last','Department','Yearly')
        ,arr_varchar2_udt('Name','Name','Name','Salary')
    );
    v_sql CLOB;
    --v_sysrefcursor SYS_REFCURSOR;
BEGIN
    v_sql := q'{WITH 
FUNCTION wget(
    p_arr   ARR_VARCHAR2_UDT
    ,p_i    NUMBER
) RETURN VARCHAR2
AS
BEGIN
    RETURN p_arr(p_i);
END;
a AS (
    SELECT rownum AS rn, t.COLUMN_VALUE AS arr 
    FROM TABLE(:header_table) t
)
SELECT wget(a.arr,1) AS c1}';

    FOR i IN 2..v_arr_headers(1).COUNT
    LOOP
        v_sql := v_sql||', wget(a.arr,'||TO_CHAR(i)||') AS c'||TO_CHAR(i);
    END LOOP;
    v_sql := v_sql||q'{
FROM a
ORDER BY rn}';

    OPEN :curs FOR v_sql USING v_arr_headers;
END;
/
print curs

And the output:

SQL> @y.sql

PL/SQL procedure successfully completed.

+                                                 +
First                    Last                     Department               Yearly
Name                     Name                     Name                     Salary

SQL>

That looks pretty good. We could use it as-is for any project and be pretty pleased.

Refactoring the Refactoring

I like this concept of opening a cursor from a two dimensional collection object enough that I added a function named get_cursor_from_collections to my PL/SQL utility package named app_csv_pkg. You can find this at on github at plsql_utilities. There is a version of get_cursor_from_collections in the package perlish_util_pkg too if you are bent that way.

Also in app_csv_pkg is a function that will parse a CLOB containing CSV data (like our column headers) and return an arr_arr_varchar2_udt object. How handy is that? Let’s see.

set pagesize 0
set trimspool on
set linesize 100
column c1 format a24
column c2 format a24
column c3 format a24
column c4 format a24
var curs REFCURSOR
DECLARE
/*
    v_arr_headers   arr_arr_varchar2_udt := arr_arr_varchar2_udt(
        arr_varchar2_udt('+',NULL,'+',NULL)
        ,arr_varchar2_udt('First','Last','Department','Yearly')
        ,arr_varchar2_udt('Name','Name','Name','Salary')
    );
*/
    v_arr_headers   arr_arr_varchar2_udt := app_csv_pkg.split_clob_to_fields(
q'{+,,+,
First,Last,Department,Yearly
Name,Name,Name,Salary}'
    );

    --v_src   SYS_REFCURSOR;
BEGIN
    --v_src := app_csv_pkg.get_cursor_from_collections(v_arr_headers);
    :curs := app_csv_pkg.get_cursor_from_collections(v_arr_headers);
END;
/
print curs

And the output:

PL/SQL procedure successfully completed.
+                                                 +
First                    Last                     Department               Yearly
Name                     Name                     Name                     Salary

SQL>

Now we’re cooking with gas!

You may already have the column headers in a spreadsheet. I often have them as part of a requirements document. Export them as CSV (Save As CSV might give you an evil windows character at the start of the file - use Export instead). Now you can copy/paste that file content directly into your PL/SQL program. It will even handle commas embedded in the fields correctly.

Conclusion

This seemed like a problem that should not be so hard in PL/SQL, but it wasn’t as easy as one would like. I like where this wound up with some utility methods to make it easier.

Versioning Oracle Code from Open Source

Sun, 13 Nov 2022 10:00:00 -0500

Introduction

My team have a moderately large installed base of Oracle packages that use Marc Bleron’s ExcelGen package. Version 3 of the package was released recently and we have use cases that can benefit from the new features. Regression testing the installed base of code as is required under corporate SDLC policy is not an attractive proposition.

The strategy we chose is to name the newly changed Oracle objects differently than the existing ones so that we can have multiple versions installed in production. For example ExcelGen package will be named ExcelGen_v3. This isn’t ideal, but all future uses, including any time a job is changed and must be regression tested anyway, will use the new version. Eventually, the existing version will be retired.

This article describes a technique and a Perl script to facilitate versioning of the changed objects. Full disclosure - initially I did much of this manually. For this article I cleaned up the Perl script to address oddities I found during the exercise and to keep my Perl skills from fading away.

Scenario

ExcelGen is open-source software hosted on GitHub. Instructions for cloning the repository point out you should use –recurse-submodules or else you must copy the two submodules down separately. The use of submodules is good technique, but complicates our task.

The repository follows a model with a separate file (or files) for every Oracle object. The files are also named the same as the Oracle objects except for the file suffix. This is best practice for source code control of Oracle artifacts in my opinion, and fortunate for the pattern I followed here.

We need to identify all files that changed since our production deployment, the objects defined in those files, and any dependent objects that would be impacted by the compile.

Identifying Changed Files in Git

We could simply run diff commands to identify files that changed, but we have a nice tool in git that keeps track of all this. In order to use it we must know the commit point of the files we already deployed. In my case I know that tag “v2.5.1” represents the files I deployed from the main ExcelGen repository. The MSUtilities repository has no changes in the last two years, and ExcelCommons change I last deployed was commit 383ff141bffd3bd31ed3054106346a948575469a. In git a tag, a branch and a commit are all simply pointers that are interchangeable for identifying the state of code at a particular point in the history.

To identify changed files from the top level directory we run the command:

#git diff --name-only prior_version_tag_or_commit
git diff --name-only v2.5.1

Discounting the test_cases, samples and resources files, it turns out the only ones impacted in the main repository are

plsql/ExcelGen.pkb
plsql/ExcelGen.pks

Changing directory to ExcelCommons we run the command for this submodule:

git diff --name-only 383ff141bffd3bd31ed3054106346a948575469a

This yields:

plsql/ExcelTableCell.tps
plsql/ExcelTypes.pkb
plsql/ExcelTypes.pks
plsql/xutl_xlsb.pkb
plsql/xutl_xlsb.pks

From this we have a list of changed Oracle objects:

ExcelGen
ExcelTableCell
ExcelTypes
xutl_xlsb

We could simply look at each one with Toad or SqlDeveloper for dependencies, but I felt like playing, so we’ll let Perl find the dependencies for us while we prepare to change all occurrences of these four identifiers by adding “_v3” to the end. Note it is not as simple as a global search and replace with your text editor because these could be sub-parts of larger names that don’t change. For example the string ExcelTableCell appears in ExcelTableCellList.

Perl Script Overkill

At a whopping 125 lines this monster outgrew my immediate need, but it does some things you may find interesting. The start of the script gathers the arguments. The first argument is the name of the install script from which it will read the names of the files we would install if this were a fresh install. After that we provide a version string to add to the object names. In this case I provided “_v3”. Finally follow the names of the objects that we know need to be versioned as a seed. The script may add more.

#!/usr/bin/env perl
use strict;
use warnings;
my $usage = "USAGE: $0 install_script version_string identifier1 [identifier2..]\nwhere 'identifier' is oracle object that you know has changed in the release.\n";
#
# In each git module and submodule, 
# "git diff --name-only prior_version_tag_or_commit"
# That gives us list of files from which we should be able to name objects 
# we need to version and install that are different than
# what we already deployed. 
# This script assumes the files are named after the objects they build.
# object_name_version.pl will also check recursively for objects in the
# install script that use changed objects as those need to be versioned too.
#
die "$0: must provide at least 3 args.\n$usage\n" unless @ARGV >= 3;
my $install_script = shift;
my $version_string = shift;
# now only idendifiters are left in @ARGV

my (@on_re_list, %on_list);
for my $on (@ARGV) { # we want unique list of lower case object names
    @on_list { lc $on } = 1;
}

Next we create subroutines that we can call multiple times as we identify dependencies. The first creates regular expressions for each versioned object that we can use both for searching and also for modifying the code. The parsing of an identifier is something I’ve done before for the vim plsql syntax file and also for the Ruby Rouge plsql lexer.

sub build_on_re_list {
  # we know the objects that changed. 
  # Find those and any files that contain them.
  # Let us construct an array of regular expressions to match on. Then
  # we can use those same regexp's to substiture the new names.
  print STDERR "calling build_on_re_list\n";
  @on_re_list = ();
  for my $identifier (keys %on_list) {
        print STDERR "adding re for $identifier\n";
        push @on_re_list, qr/
    (                                       # Capture before identifier string if any
            ^                               # could be matching at start of line
        |
            [^a-z0-9_#\$]                   # any character that is not an identifier
    )
    (                                       # capture our object word
        $identifier
    )
    (                                       # capture character after our object word
            [^a-z0-9_#\$]                   # any character that is not an identifier
        |
            $                               # could be end of line
    )
/imx;
    # ignore case, ignore whitespace in RE, match ^$ at newline
  }

  # the replacement string will be "${1}${2}${version_string}$3"
}

This subroutine can be called two ways. The first to identify impacted files and the second option creates new files named with the version string that we can deploy.

# sub find_files uses these so set before defining
my ($install_script_out);
($install_script_out = $install_script) =~ s/\.sql/${version_string}.sql/;
open(OFL, ">", "$install_script_out") or die "failed open $install_script_out for write: $!";

sub find_files {
    # look through files for references to the objects that changed
    my ($look_only) = @_ > 0 ? $_[0] : 0;
    print STDERR "calling find_files with look_only=$look_only\n";
    my @file_names;
    open(FL, "<", $install_script) or die "failed to open install script $!";
    while (<FL>) {
        next unless /^@/;
        chomp;
        my ($fn) = $_ =~ /@@?(.+)/;
        print STDERR "checking $fn\n";
        my $data;
        {
            open my $f, '<', $fn or die;
            local $/ = undef; # slurp mode
            $data = <$f>;
            close $f;
        }
        my $match_file;
        for my $re (@on_re_list) {
            if ($data =~ $re) {
                $match_file = $fn;
                last;
            }
        }

        next unless $match_file; # we did not find any interesting identifiers

        # reduce file name to oracle identifier. no slash or dot, followed
        # by last dot and non-dots to end of string.
        my ($id) = $match_file =~ m!([^./]+)\.[^.]+$!i; 
        $on_list{ lc $id } = 1;
        
        next if $look_only;     
        # if we are done recursively looking for all objects we need to change
        # continue on to making a new version of the files needed.

        # need to change all occurences of versioned objects inside this file
        for my $re (@on_re_list) {
            $data =~ s/$re/${1}${2}${version_string}$3/g;
        }
        # write the versioned file out with new name
        my $ofname;
        # new output file name has version string before dot
        ($ofname = $match_file) =~ s/\.(.*)/${version_string}.$1/; 
        print STDERR "found changed object names in ${fn}. Creating $ofname\n";
        open OF, '>', $ofname;
        print OF $data;
        close OF;
        # add it to the new install script
        print OFL '@@', $ofname, "\n";
    }
} # find_files

We call the subroutines repeatedly to add to the object lists as dependencies are found. When we find no new ones we exit the loop and call the version that creates our new files. We also write out a new install script with only the objects that we change.

# keep looking recursively for more objects we may need to version because
# of dependencies until we don't find any new ones
do {
    build_on_re_list();
    find_files(1); # could add some to on_list
} while (@on_re_list < keys %on_list) 
;

# Now call it so that it creates the versioned files
find_files(); 

print STDERR "writing $install_script_out\n";
close OFL;

Execute

[C:/Users/Leeli/Documents/ExcelGen] (master)
$ perl -w object_name_version.pl install.sql _v3 excelgen exceltablecell exceltypes xutl_xlsb

Output:

calling build_on_re_list
adding re for exceltypes
adding re for exceltablecell
adding re for excelgen
adding re for xutl_xlsb
calling find_files with look_only=1
checking MSUtilities/CDFManager/xutl_cdf.pks
checking MSUtilities/CDFManager/xutl_cdf.pkb
checking MSUtilities/OfficeCrypto/xutl_offcrypto.pks
checking MSUtilities/OfficeCrypto/xutl_offcrypto.pkb
checking ExcelCommons/plsql/ExcelTableCell.tps
checking ExcelCommons/plsql/ExcelTableCellList.tps
checking ExcelCommons/plsql/ExcelTypes.pks
checking ExcelCommons/plsql/ExcelTypes.pkb
checking ExcelCommons/plsql/xutl_xlsb.pks
checking ExcelCommons/plsql/xutl_xlsb.pkb
checking plsql/ExcelGen.pks
checking plsql/ExcelGen.pkb
calling build_on_re_list
adding re for exceltablecelllist
adding re for excelgen
adding re for xutl_xlsb
adding re for exceltypes
adding re for exceltablecell
calling find_files with look_only=1
checking MSUtilities/CDFManager/xutl_cdf.pks
checking MSUtilities/CDFManager/xutl_cdf.pkb
checking MSUtilities/OfficeCrypto/xutl_offcrypto.pks
checking MSUtilities/OfficeCrypto/xutl_offcrypto.pkb
checking ExcelCommons/plsql/ExcelTableCell.tps
checking ExcelCommons/plsql/ExcelTableCellList.tps
checking ExcelCommons/plsql/ExcelTypes.pks
checking ExcelCommons/plsql/ExcelTypes.pkb
checking ExcelCommons/plsql/xutl_xlsb.pks
checking ExcelCommons/plsql/xutl_xlsb.pkb
checking plsql/ExcelGen.pks
checking plsql/ExcelGen.pkb
calling find_files with look_only=0
checking MSUtilities/CDFManager/xutl_cdf.pks
checking MSUtilities/CDFManager/xutl_cdf.pkb
checking MSUtilities/OfficeCrypto/xutl_offcrypto.pks
checking MSUtilities/OfficeCrypto/xutl_offcrypto.pkb
checking ExcelCommons/plsql/ExcelTableCell.tps
found changed object names in ExcelCommons/plsql/ExcelTableCell.tps. Creating ExcelCommons/plsql/ExcelTableCell_v3.tps
checking ExcelCommons/plsql/ExcelTableCellList.tps
found changed object names in ExcelCommons/plsql/ExcelTableCellList.tps. Creating ExcelCommons/plsql/ExcelTableCellList_v3.tps
checking ExcelCommons/plsql/ExcelTypes.pks
found changed object names in ExcelCommons/plsql/ExcelTypes.pks. Creating ExcelCommons/plsql/ExcelTypes_v3.pks
checking ExcelCommons/plsql/ExcelTypes.pkb
found changed object names in ExcelCommons/plsql/ExcelTypes.pkb. Creating ExcelCommons/plsql/ExcelTypes_v3.pkb
checking ExcelCommons/plsql/xutl_xlsb.pks
found changed object names in ExcelCommons/plsql/xutl_xlsb.pks. Creating ExcelCommons/plsql/xutl_xlsb_v3.pks
checking ExcelCommons/plsql/xutl_xlsb.pkb
found changed object names in ExcelCommons/plsql/xutl_xlsb.pkb. Creating ExcelCommons/plsql/xutl_xlsb_v3.pkb
checking plsql/ExcelGen.pks
found changed object names in plsql/ExcelGen.pks. Creating plsql/ExcelGen_v3.pks
checking plsql/ExcelGen.pkb
found changed object names in plsql/ExcelGen.pkb. Creating plsql/ExcelGen_v3.pkb
writing install_v3.sql

The resulting install script “install_v3.sql” is

@@ExcelCommons/plsql/ExcelTableCell_v3.tps
@@ExcelCommons/plsql/ExcelTableCellList_v3.tps
@@ExcelCommons/plsql/ExcelTypes_v3.pks
@@ExcelCommons/plsql/ExcelTypes_v3.pkb
@@ExcelCommons/plsql/xutl_xlsb_v3.pks
@@ExcelCommons/plsql/xutl_xlsb_v3.pkb
@@plsql/ExcelGen_v3.pks
@@plsql/ExcelGen_v3.pkb

This install assumes other base objects were already installed and are neither changed nor impacted.

Conclusion

This is overkill for the task. The regular expression for replacing object names is super useful, but much of the rest of this task could have been (and was) completed manually this time. I can see a day when it would be a much bigger effort though, and the exercise was useful to maintain my skills. Maybe you will see something useful to you.

Tuning Query with OR Conditions in a NOT EXISTS

Sun, 09 Oct 2022 02:00:00 -0400

Introduction

While tuning an Oracle query provided by a business user I ran into a roadblock that confounded me. The query contained a NOT EXISTS clause against a large row set, one that was a Common Table Expression (CTE or WITH clause) that was relatively complex. The CTE plan was fine, but it resulted in about a million rows. An ideal plan to satisfy the NOT EXISTS from my perspective would use a HASH JOIN ANTI against that row set. That was not happening. It was doing a FILTER operation. I’m not 100% sure what Oracle is doing under the covers on the FILTER.

Problem

The problem SQL is too complex and intertwined with my client’s business to reproduce here, but pseudo code is good enough for the discussion.

WITH ne AS (
    SELECT 
        lookup_value 
    FROM large_result_set L
    WHERE some_fancy_condition = 'TRUE'
)
SELECT --+ cardinality(a 1000000)
    my_key_value, my_lookup_name, column_list
FROM another_large_result_set a
WHERE 
    -- this is done using an index one by one which is ok
    NOT EXISTS (
        SELECT 1
        FROM some_other_table s
        WHERE s.key_value = a.my_key_value
    )
    -- below is the problem antijoin
    AND NOT EXISTS (
        SELECT 1
        FROM ne
        WHERE ne.lookup_name = a.my_lookup_name
            -- this DECODE represents an OR condition
            AND DECODE(ne.lookup_value, '*', 1, a.my_key_value, 1, 0) = 1
            /* --I would prefer this syntax
                AND (ne.lookup_value = '*' OR ne.lookup_value = a.my_key_value)
            */
    )

The picture below (Toad tree view) from the actual plan shows a FILTER operation with 3 components - another_large_result_set, some_other_table, and ne. We can tell from the fact that it shows the index lookup, that it is doing a one by one index probe of some_other_table. Less obvious what it is doing with ne, but in the absence of other evidence, we must assume it is walking through the heap in memory looking for a match. Maybe it has sorted it and is doing something smarter than that, but it is not a hash.

Figure 1 - Explain Plan of Filter Operation for Antijoin with OR

Why could it not hash the ne row set and probe it with a HASH JOIN ANTI? Because you cannot hash an OR condition.

If it is a regular join instead of an anti-join that has this OR condition, the optimizer can break the problem into multiple HASH JOIN’s and do a CONCATENATION operation of the results for each of the OR conditions.

I do not recall seeing a plan where the optimizer chooses two separate JOIN ANTI operations in series. It would not be a CONCATENATION operation between them, but the opposite of that because they are anti-joins. I don’t think the Oracle optimizer has an operation for splitting an JOIN ANTI into two JOIN ANTI’s in sequence.

Solution

I first thought maybe the DECODE function on the value, being a function, was confusing the optimizer. Rewriting that in my preferred syntax with an OR condition rather than the implied OR of the DECODE did not solve the issue. It was a long shot given my understanding of the optimizer, but I don’t know everything, the optimizer is constantly evolving, and I wanted to give it the best chance to solve the issue.

I’ve seen this before on regular joins with a similar construct of a lookup table having wild cards. In that case it was multiple conditions in the join with wild cards optional on all of them. The optimizer was overwhelmed and just did a giant filter as we saw with this anti-join. One would think it might be able to break the problem into multiple joins with concatenation, but the multiple join conditions seems to have exceeded the optimizer’s breadth of options.

I recall having to break the problem down into two separate joins manually, one for the wild card and one for the direct match of the key, at least for the first of multiple wild-carded join conditions. That at least was able to reduce the join set to one that could be filtered for the remaining conditions.

I tried a similar trick here rewriting the query as two separate NOT EXISTS with an AND condition between them.

WITH ne AS (
    SELECT 
        lookup_value 
    FROM large_result_set L
    WHERE some_fancy_condition = 'TRUE'
)
SELECT --+ cardinality(a 1000000)
    my_key_value, my_lookup_name, column_list
FROM another_large_result_set a
WHERE 
    -- this is done using an index one by one which is ok
    NOT EXISTS ( -- not exists 0
        SELECT 1
        FROM some_other_table s
        WHERE s.key_value = a.my_key_value
    )
    AND NOT EXISTS ( -- not exists 1
        SELECT 1
        FROM ne
        WHERE ne.lookup_name = a.my_lookup_name
            AND ne.lookup_value = '*' 
    ) AND NOT EXISTS ( -- not exists 2
        SELECT 1
        FROM ne
        WHERE ne.lookup_name = a.my_lookup_name
            AND ne.lookup_value <> '*' 
            AND ne.lookup_value = a.my_key_value
    )

Figure 2 - Explain Plan of Filter Operation for Antijoin no OR

That worked swimmingly. The optimizer hashed all of the rows from ne with a lookup value of ‘*’, also hashed all of the rows from ne with at lookup value that was not ‘*’, and did a HASH JOIN ANTI against each in pipelined series. This was a much more efficient plan that will also scale well as the number of rows increases over time. The actual run time was about a third what it was before the re-write. Honestly I expected better than that, so maybe Oracle is doing some form of optimization on that filter operation under the covers.

It is still showing the antijoin of some_other_table as a FILTER. I do not know why this presents as a filter rather than a NESTED LOOP JOIN ANTI. I’ve seen nested loop anti-join be shown by the optimizer rather than filter in other situations and am unsure what the difference is. I went trolling through Jonathan Lewis’s fine book Cost-Based Oracle Fundamentals, but did not find an example of a plan using FILTER. It may be something that Oracle added after 10g which is the release the book covered.

Conclusion

When dealing with OR join conditions (IN lists are OR conditions too), there is only so much the optimizer can do. When you are getting an unacceptable plan for a query with ORs in the join conditions, or especially anti-join conditions, consider how you might be able to rewrite the query with two joins, one for each of the ORs.

Manipulating XLSX Spreadsheets in PL/SQL

Sun, 18 Sep 2022 07:30:00 -0400

Introduction

There are many common methods for inputting and outputting spreadsheet data from Oracle. Below are two limited lists for talking purposes. You may well have others.

Our use case is to supplement the data in a user maintained spreadsheet using data from an Oracle database. We must as faithfully as possible maintain the existing data in the spreadsheet while adding new columns.

app_read_xlsx_udt Use Case Diagram

One limitation shared by common XLSX/Oracle methods is the concept that a column may contain only one datatype. A date column contains only dates or NULLs. A numeric column contains only numbers or NULLS. Yet, spreadsheet data cells may contain any kind of data and formatting can be specific to the cell. We traditionally think of columns in the spreadsheet of being a single datatype and formatting is typically accomplished at the column level, but our intrepid business user may place a string such as “Terminated” or “N/A” in a column that otherwise has Date values.

If we were tasked with replacing their entire business process, we would of course normalize this data and separate the non-date content into a separate column; however, our short term task is to supplement the data the business uses in an existing process. We do not control their process or their data.

How can we maintain the polymorphic cell content of the spreadsheet and replicate it on output?

Content

Example Methods for Manipulating XLSX with Oracle
- Input Spreadsheet to Oracle
- Output Spreadsheet from Oracle
Option Elimination
- A Microsoft Option
- Oracle Specific Options
ExcelGen and ANYDATA Columns
app_read_xlsx
Conclusion

Example Methods for Manipulating XLSX with Oracle

Input Spreadsheet to Oracle

An ODBC connection from Excel to insert spreadsheet content into Oracle table - perhaps using a VB Macro
Save as CSV and load with external table or sqlldr
Toad, SQL Developer or other client tool to read xlsx and extract into an Oracle table
An ETL tool like Informatica which can read XLSX
A PL/SQL tool that can parse XLSX such as Anton Scheffer’s as_read_xlsx or Marc Bleron’s ExcelTable.

Output Spreadsheet from Oracle

An ODBC connection from Excel to read a query resultset - perhaps using a VB Macro
Output a CSV text file from Oracle and open with Excel
Toad, SQL Developer or other client tool generate an XLSX file from a query resultset
An ETL tool like Informatica which can generate XLSX
A PL/SQL tool to generate XLSX directly in the database like Marc Bleron’s ExcelGen
Python xlsxwriter

Option Elimination

A Microsoft Option

The most full featured option for this use case is the Microsoft Excel library available from Visual Basic or C#. One can create a macro directly inside the spreadsheet using an ODBC connection to the database to perform the task. Limitations are:

The user’s personal Oracle login must have access to all needed queries and functions. Since the user’s personal login is not tracked as an IT asset, care must be taken to encapsulate all necessary access via appropriate roles. Even so, maintenance of user accounts and associating the correct roles to include access needed by this spreadsheet becomes a burden.
The code lives on a User’s desktop or maybe a shared drive. Perhaps the gold copy could be kept under source code control and made available to the users, but it doesn’t really fit well with common SDLC practices. The business could also change the spreadsheet parts that are not related to the data pull. At that point it is out of sync with the IT maintained copy.
When the user experiences problems, the first thing IT must question is whether the user is employing the latest version of the code.
The expertise to perform this task, though not uncommon, is not necessarily in the toolbox for a journeyman Oracle developer.

An extension of this option may be to use a Sql Server database (or whatever it is called now) to host the operation on the spreadsheet. I have not done this and do not know limitations, but my understanding is these Excel libraries should be available to code that is run from the database. DBLink accounts that can connect to the Oracle database can be maintained for this server and the code can be put under source code control. This added complexity does not appeal to me, but a shop who already run Sql Server databases, especially those supporting batch operations and/or existing user interaction screens that include file transfer may favor this option.

Oracle Specific Options

As mentioned, most of the listed options stumble when faced with a column that can contain cells with different data types. We can reduce everything to text, but then when we output to a spreadsheet again, we’ve lost information and our business partner is not pleased. This eliminates using CSV or a client like Toad, SqlDeveloper or Informatica to deliver the data to the database.

We are left with as_read_xlsx or ExcelTable for the input, both of which can deliver the raw spreadsheet data at the cell level with data type intact.

Assuming we can maintain the polymorphic content information about individual cells on input, we face the same limitation on output. A tool like Toad, SQL Developer or Informatica reads resultsets from queries. I do not believe any of them have been enhanced to support the Oracle ANYDATA object type so we are restricted to cell data that is either character, date or number.

I mention Python xlsxwriter, but I really don’t know if it has such a capability. What I have seen of it in practice is that it takes a resultset from a query similarly to the others.

I set up this straw man with a non-exhaustive list of options, so there may well be another choice out there. I would love to hear about it if so. The choice I made is to use ExcelGen to create the output spreadsheet.

Of the two possible input tools, ExcelTable is more sophisticated and already supports providing the raw cell data as ANYDATA values. It has other functions that return ANYDATASET results which is fantastic, but these require you to provide the column type and header information in your code. I badly wanted to use the first row of the spreadsheet for the column headers and do not want to specify the data type. I considered adding functionality to do so, but was not encouraged. Perhaps I did not sell it well enough.

Although I wanted to use ExcelTable, the more I looked into it the bigger the sinking feeling I had that it was beyond a level of complexity I felt I could leave with my employer. Without community support it was not an option.

By process of elimination we are left with as_read_xlsx which is a perfectly serviceable tool that is in widespread use.

ExcelGen and ANYDATA Columns

The currently published version of ExcelGen does not support ANYDATA data type input columns. I created a fork and Pull Request adding this functionality, but was not aware that Marc was in the midst of a refactor/redesign of ExcelGen and he politely declined the PR. He agreed that allowing ANYDATA input columns was useful and he liked the idea of supporting it. I’ll call that a soft commitment. If the next version of ExcelGen does not support ANYDATA input, I’ll create another pull request to add it. I’m fairly confident it will be included in a future release one way or another.

Meanwhile, to implement the solution discussed here you will need to use my forked version of ExcelGen.

app_read_xlsx

Although it uses as_read_xlsx as the underlying workhorse, app_read_xlsx takes care of many of the details needed to treat a spreadsheet as a row source, using the first row data values as the column names and output column headers.

The documentation found in the README.md at the above link discusses some of what is covered in this article. It also contains what I think is a decent example showing the problem and the solution. I am not going to repeat those here but am going to borrow the How it Works section from that document.

Overview

The input datastream from as_read_xlsx is a table of cell data. The ordinal row and column numbers of the spreadsheet are columns/attributes in this data stream.
Empty cells are not present in the data.
The concept of column headers and database identifiers for the columns is not present in this structure.
Each cell is represented with a polymorphic structure containing a cell_type attribute and a value in one of the attributes string_val, number_val, date_val, formula. formula is out of scope for this implementation. Our design pattern converts this polymorphic structure into an Oracle ANYDATA object type.

Presenting the cell data in a two dimensional standard database pattern requires

extract column identifiers and number of columns from the first row of the input data
pivot spreadsheet columns into rows
densify the missing/empty cells
convert multi-attribute polymorphic cell representation into ANYDATA objects
present the ANYDATA cell objects in standard database TABLE structure with rows and columns named from the spreadsheet column headers

Doing this requires a runtime determination of the resultset type. It is not difficult to do this for a PL/SQL cursor as we can use a weakly typed SYS_REFCURSOR. It is much harder to present the results to the SQL engine in a way that the resultset may be joined and extended.

Considerations

When one hears the term polymorphic resultset, we instantly turn to the cool new Oracle toy (well, new as of Oracle 18c) of Polymorphic Table Functions. Unfortunately, this design pattern only supports standard Oracle datatypes. Object types such as ANYDATA are not supported, at least as of Oracle 19c.

Another method for achieving this is the ANYDATASET technique which is built with ANYTYPE. Building these requires producing ODCI level code, whether in PL/SQL or another compiled language such as Java or C. Although this pattern can be followed reasonably well at a cookbook level for standard data types with a moderate level of study, extending it to handle piece-wise construction of complex object types such as the ANYDATA objects is non-trivial. (see ExcelTable.getRows in ExcelTable for an example of using ANYDATASET with standard datatypes.) This is a level of complexity the author has seldom observed within most corporate IT departments. If there were community support of this I would be willing, but for this project it exceeds the complexity level with which I’m comfortable encumbering my current employer.

The level of complexity I settled on was using a compile time known object type representing a row, and standard pipelined table function returning a collection of that row object type. This is a well known and documented technique that should be in the wheelhouse of most journeyman level Oracle practitioners. The only slightly tricky part I added was the use of a nested table collection inside this object and an object method named get for extracting members of that nested table in a SQL statement.

Technique

We start with a collection object type of ANYDATA objects.

CREATE OR REPLACE TYPE arr_anydata_udt FORCE AS TABLE OF sys.anydata;
/

Next we build an object type that can be piped from our table function:

CREATE OR REPLACE TYPE app_read_xlsx_row_udt FORCE AS OBJECT (
    data_row_nr NUMBER
    ,aa         arr_anydata_udt
    ,MEMBER FUNCTION get(p_i NUMBER) RETURN SYS.anydata
);
/
CREATE OR REPLACE TYPE BODY app_read_xlsx_row_udt AS
    MEMBER FUNCTION get(p_i NUMBER)
    RETURN SYS.anydata
    AS
    BEGIN
        RETURN aa(p_i);
    END get;
END;
/

The get method is necessary to access a member of the nested table collection from within SQL (inside PL/SQL you could just use aa(i)).

Then to be able to define our pipelined table function we need a nested table type of these elements:

CREATE OR REPLACE TYPE arr_app_read_xlsx_row_udt FORCE AS TABLE OF app_read_xlsx_row_udt;
/

Our pipelined table function (which is a static method of our main object type app_read_xlsx_udt) can then be declared as:

    STATIC FUNCTION get_data_rows(
         p_ctx      NUMBER
        ,p_col_cnt  NUMBER
    ) RETURN arr_app_read_xlsx_row_udt PIPELINED

This still leaves the task of generating a SQL select list that turns the collection elements aa.get(i) into columns with an identifier based on the first row of the spreadsheet. That is done by calling the get_sql method of app_read_xlsx_udt. It builds a dynamic SQL statement for you that you can then use as part of a larger application level SQL statement as shown in the examples section and reproduced here.

SELECT X.R.data_row_nr AS data_row_nr,
   X.R.get(1) AS "id"
    ,X.R.get(2) AS "data"
    ,X.R.get(3) AS "ddata"
  FROM (
    SELECT VALUE(t) AS R -- full object, not the object members * would provide
    FROM TABLE(LEE.app_read_xlsx_udt.get_data_rows(1,3)) t
  ) X

Code Dive

The manual page for app_read_xlsx_udt constructor describes how the output from as_read_xlsx is read into a global temporary table (GTT). Subsequent reads of this data for the first row determine the number of columns and column headers to populate the object attributes during the constructor call. After that we read the row data from the GTT using the static pipelined table function app_read_xlsx_udt.get_data_rows.

This code is a little interesting in how it uses a restricted package ( ACCESSIBLE BY (app_read_xlsx_udt) ) to maintain a set of session specific context numbers allowing for multiple spreadsheets to be read simultaneously in a single session. The package also implements the call to as_read_xlsx when it populates the GTT. You can look at your leisure.

In Overview we listed the tasks the function needed to perform in a bullet list. Most of these tasks are handled in the static pipelined table function app_read_xlsx_udt.get_data_rows via a cursor that uses a bind variable for the number of columns we have in our input (gathered during object constructor call).

These two bullet items are covered by the first code section:

densify the missing/empty cells
convert multi-attribute polymorphic cell representation into ANYDATA objects

First, generate a Common Table Expression (CTE or WITH Clause view) consisting of an integer for each column.

        CURSOR c_filled_gaps IS
WITH cols AS (
    SELECT level AS col_nr FROM dual CONNECT BY level <= p_col_cnt
)

Next, grab our data from the GTT, but only the rows and columns of interest.

, this_ctx_cols AS (
    SELECT row_nr, col_nr, cell_type, string_val, date_val, number_val
    FROM as_read_xlsx_gtt
    WHERE ctx = p_ctx AND row_nr > 1 AND col_nr <= p_col_cnt
)

The CASE statement below examines the data type of the cell and calls the appropriate static constructor for an ANYDATA object. Notice the funky method for creating a NULL ANYDATA object. If you find a less kludgey method, let me know.

The PARTITION BY and RIGHT OUTER JOIN are an Oracle technique I do not know the name of for densifying data. Heck, that could be the official name of the technique. It makes sure that on any given row_nr we selected from the GTT, there will be a row in the output for every column.

, ad_cols AS (
    SELECT t.row_nr, cols.col_nr
        ,CASE t.cell_type
            WHEN 'S' THEN SYS.ANYDATA.convertVarchar2(t.string_val)
            WHEN 'D' THEN SYS.ANYDATA.convertDate(t.date_val)
            WHEN 'N' THEN SYS.ANYDATA.convertNumber(t.number_val)
            ELSE SYS.ANYDATA.convertVarchar2(NULL) -- must have a placeholder for collect
        END AS ad
    FROM this_ctx_cols t
    PARTITION BY (t.row_nr) -- fill gaps for empty cells
    RIGHT OUTER JOIN cols
        ON cols.col_nr = t.col_nr
)

Now that we are promised a row for every cell, we can use a COLLECT aggregation function to build a nested table collection object in SQL. That takes care of pivoting the column data into a single row per spreadsheet row, if not exactly pivoting into columns yet.

pivot spreadsheet columns into rows

To complete the cursor query we plug these values into the default constructor for our app_read_xlsx_row_udt object described earlier. That is what our function must PIPE ROW out.

, ad_arr AS (
    SELECT row_nr - 1 AS data_row_nr
        ,CAST( COLLECT(ad ORDER BY col_nr) AS arr_anydata_udt) AS vaa
    FROM ad_cols
    GROUP BY row_nr
)
SELECT app_read_xlsx_row_udt(data_row_nr, vaa)
FROM ad_arr
;

The rest of the function is simple boilerplate for a pipelined table function:

        OPEN c_filled_gaps;
        LOOP
            FETCH c_filled_gaps BULK COLLECT INTO v_arr LIMIT 100;
            EXIT WHEN v_arr.COUNT = 0;
            FOR i IN 1..v_arr.COUNT
            LOOP
                PIPE ROW(v_arr(i));
            END LOOP;
        END LOOP;
        CLOSE c_filled_gaps;
        RETURN;

No matter how many columns we have or what our column names are, we have a single compile-time representation of our resultset coming from this pipelined table function. With a little help from member function app_read_xlsx_udt.get_sql we can get our columns out with proper names using regular SQL and use it as if it had come from a table.

You can look at the code for get_sql at your leisure, but the example shown in the Technique section and repeated next is the best way to understand it:

SELECT X.R.data_row_nr AS data_row_nr,
   X.R.get(1) AS "id"
    ,X.R.get(2) AS "data"
    ,X.R.get(3) AS "ddata"
  FROM (
    SELECT VALUE(t) AS R -- full object, not the object members * would provide
    FROM TABLE(LEE.app_read_xlsx_udt.get_data_rows(1,3)) t
  ) X

I’ll mention something that confounds those who have not used Oracle objects often. In order to access a member method or element of an object you must use a table alias as part of the name. One cannot rely on an actual table name or an implied (no) table name; it must be a table alias. In this case the table alias is ‘X’.

The other thing to mention is the use of the VALUE function. The most common way we extract data from a pipelined table function is via SELECT *. If we do that here, we will get the elements of the app_read_xlsx_row_udt object rather than the object itself. We need the get member method of the object, so it is important that we retrieve the object intact rather than the object elements. VALUE gives us the actual object returned from the pipelined table function rather than the object elements. It’s relatively obscure.

Conclusion

This turned out to be a much harder problem than I thought it would be when I started. Limitations of Polymorphic Table Functions and the complexity of ANYTYPE/ANYDATASET took me off guard. It would have been easiest to craft one-off solutions for each spreadsheet by writing a custom cursor, but it felt like the wrong answer. Creating app_read_xlsx was the result. It is a bit messier than I would like in the way it gives you a SQL statement for your program to incorporate, but I feel it is an appropriate level of abstraction given the requirements and limitations.

Hope you find both the tool and the journey to get here helpful.

Using Perl DBD::Oracle to write LOB content

Sat, 30 Apr 2022 10:30:00 -0400

Introduction

We have a need to get large volumes of data and/or BLOB/CLOB content out of the database and onto disk on a remote client machine.

We can do it from sqlplus. Even BLOB data can be output as I demonstrated in Extracting BLOB from Oracle with Sqlplus. I happen to think it is a clunky way to do things when there are nice scripting languages around, but the one constant you can count on having available on a client is sqlplus and some form of shell script capability, even if it is Powershell on Windows.

If you have access to Pro-C compilation, there is a nice tool Tom Kyte published long ago named flat you can find here. It is a super-fast and efficient solution. It doesn’t do CLOB or BLOB, but with a little elbow grease I’m sure one could manage it. Pro-C has everything you need to handle a LOB Locator and retrieve the content.

I know it is doable in Java, and I suspect most other languages that have database connection libraries.

sqlCL can do it as described in Using SQLcl to write out BLOBs to files in 20 lines of js; however, as I’ve mentioned in other posts, sqlCL is strangely lacking on every ETL server I’ve had the pleasure of visiting. For that matter the security folk aren’t crazy about having java client programs on servers. I think they overreacted to a security issue from many years ago and have never gotten over it, but it is what it is.

This article is about extracting CSV flat files, CLOB’s and BLOB’s from Oracle using Perl.

Do You Have Perl DBD::Oracle?

You may not. It does not ship with any OS I know of other than Oracle’s own version of RHL. Neither is it available in any YUM repository I could find, so your Unix Admin is not going to be able to install it easily. They may balk at doing it at all depending on how strict the rules are under which they work.

That said, it has gotten easier as I demonstrated in Installing Perl DBD::Oracle on RHL. There is a good chance your Unix Admin isn’t allowed to alter the vendor Perl. If that is the case you may need to install your own Perl and add DBD::Oracle to that. Perhaps the Unix admin can do that for you.

I don’t mean to make light of this, but if you are going to have a useful scripting environment, whether it be Perl, Python, or something else, at some point the security mavens and corporate rule makers need to give you one and it needs to have the database connect libraries linked in. “Of course that’s true” you are thinking. Surprise! You may find the security mavens and IT infrastructure folk are not sympathetic to your need for a decent scripting environment with a built-in Oracle connection library on the ETL server. At one company I encountered a mindset that shell and sqlplus, along with a vendor ETL tool, were all you need. It was a large, sophisticated organization too. They were responding to pressures from senior management that resulted in what I would call unintended consequences.

Running Data Through Perl

Perl is fantastic as a scripting language. But like all scripting languages it has some overhead costs. It isn’t the bytecode, which is pretty efficient. The overhead is in the data structures which are fat pigs. If you look at the underlying structure of a Perl scalar variable, you will find multiple pointer and length members. For example it has a place to store an integer, a floating point number, a string, an array, and something called “magic” among other things. There is a lot of wasted space for any given scalar object.

Under normal circumstances that hardly matters. If you are running a fetchrow_array operation over a large number of rows on a big SQL query, you are creating and tearing down a scalar for every column on every row. It can really add up.

Yeah, don’t do that. You would be better off spooling it out from sqlplus. On the other hand, if you are reading and writing large chunks of data through Perl, the overhead is negligible. The underlying data movement is all handled in tight C code and is efficient.

Using DBD::Oracle to Read CLOB/BLOB Data

In the article The Ubiquitous CSV File I described multiple ways for generating CSV data and getting it out of the database. One of those ways was for the use case that you were doing it for a business user who just turns around and loads it to a spreadsheet. You can up your game by producing an XLSX file directly from the database using ExcelGen. The output of that is a BLOB.

I also have a CSV file generator in package named app_csv_pkg available in plsql_utilities. I described how it it works in Polymorphic Table Function (PTF) for CSV (take 3) The output of that is a CLOB. Well, you can take the output as single column rows in a fetch, but you still have to stream the data out somehow.

Given that you have a procedure that can produce a BLOB or CLOB with everything you need to put in the file on the client, how can we get it from the database? Let’s walk through an example.

This first part is boilerplate you will have in all of your scripts unless you have refactored it into a connection object, perhaps one that handles the password management.

We need the Oracle Type definitions, thus the extra argument to use DBD::Oracle.

#!/usr/bin/env perl
use DBI;
use DBD::Oracle qw(:ora_types);
use strict;
use warnings;

my $dbh = DBI->connect('dbi:Oracle:', 'lee@rhl1pdb', 'my secret password'
                        , {RaiseError => 1, AutoCommit => 0, RowCacheSize => -102400, ora_module_name => 'Perl' }
                      ) or die "Database connection not made: DBI::errstr";
$dbh->do("alter session set nls_date_format = 'mm/dd/yyyy'");

Next we prepare our PL/SQL anonymous block. This could have been a call to ExcelGen or your own procedure that produces a BLOB or CLOB. In this case I’m demonstrating with app_csv_pkg where we pass it a query to run as a string. It executes the query, fetches the data, converts it into CSV rows, and concatenates them into a CLOB. It then returns the CLOB as an OUT parameter.

The local hash with ora_auto_lob setting to false is so that we get back a LOB locator rather than letting the driver convert the CLOB into one giant string. We might not want to put that much data into memory, plus we have to know in advance how big it could be and set some variables to allow for it. Search the DBD::Oracle perldoc for CLOB and read all about it. Unfortunately, it is a pretty big topic.

my $sth = $dbh->prepare(q!
    BEGIN
        app_csv_pkg.get_clob(
            p_sql           => :sql
            ,p_clob         => :clob
            ,p_rec_count    => :rec_count
        ); 
    END;
!
    , { ora_auto_lob => 0 } # this says pass LOB locator, not entire lob
);

Next we create a variable with our query and 2 variables we will bind as OUT parameters. The bind options are important. The SQLT_CHR type will convert a perl string to the proper type for the CLOB input parameter up to about 2MB in size. If your query is bigger than that, you are doing something wrong. If you really need to input a CLOB, read the DBD::Oracle perldoc.

The output parameter type ORA_CLOB in this case means we will get back a LOB locator. If in the prepare statement above we had allowed ora_auto_lob to be the default TRUE, we would get back a string (as long as it wasn’t too big).

rec_count is just a number output parameter. After it runs we can find out how many records the query returned and were placed in our CLOB. Useful for logging.

my $sql = 'SELECT * FROM v$reserved_words';
my ($rec_count, $clob);

$sth->bind_param( ':sql', $sql, {ora_type => SQLT_CHR } ); # This type converts to CLOB on input up to 2MB which is plenty
$sth->bind_param_inout( ':clob', \$clob, 0, {ora_type => ORA_CLOB } ); # will be a CLOB locator
$sth->bind_param_inout( ':rec_count', \$rec_count, 0);
$sth->execute;

After the execute, $rec_count has the number of rows which we can report and $clob has a LOB locator (which is only good to use during this transaction; a commit or rollback destroys the underlying LOB.)

Now we write out the data. We are writing to STDOUT here, but you could have opened a named file and be writing to that.

Rather than bring back the entire CLOB into one huge chunk of memory, we will read and write it in pieces. Each of these is a round trip to the database so do not make them too small, but neither do we want to use so much memory on both sides of the connection that it is an issue. Just like where Oracle has a rule of thumb that 100 rows is a good bulk fetch size, the right answer is probably below 1MB and more than 10K. I don’t really know where the sweet spot is without some experimentation. Here I picked 256K bytes.

print STDERR $rec_count, " rows returned in clob\n";

# 256K chunks are not that much memory, but still big enough perl scalar creation/destruction not an issue
my $chunk_size=256*1024; 
my $offset = 1; # starts at 1, not 0
while(1) {
    my $data = $dbh->ora_lob_read( $clob, $offset, $chunk_size );
    last unless length $data;
    print $data;
    $offset += $chunk_size;
}

$dbh->rollback; # to end the transaction concerning temp lob locator and freeing it so perl destructor doesn't complain

I’m not going to print out the results, but this works just fine. My sample query output is not bigger than the chunk size, so it didn’t loop at first. I had to make it smaller and put in some debug prints to prove it, but it works.

Conclusion

I’ve waved my hands around a lot in prior articles saying you can get CLOB and BLOB data out on the client. I showed it with sqlplus and sqlCL. I have used Tom Kyte’s flat utility for years (though the security mavens don’t want me near a C compiler and Devops won’t give me a deploy method for it). It was time to put up some proof that we can do it all with a scripting language efficiently too. Here ya’ go.

Hope this helps.

Syntax Highlighting for PL/SQL in vim

Fri, 29 Apr 2022 12:30:00 -0400

Introduction

In the blog post A Ruby/Rouge Lexer Class for Oracle PL/SQL I describe how I wound up creating a PL/SQL lexer in Ruby-Rouge so that I could make the PL/SQL code blocks in my posts look good.

I tried to apply my color scheme to vim but quickly ran into trouble. The PL/SQL syntax file for vim had not been updated since Oracle 9i and it had flaws such as not supporting q-quote operator. It also did not have folding.

Previously having done work to get all of the 19c keywords straight, and familiar with what needed to be done to parse PL/SQL and SQL, I applied what I knew to the existing plsql.vim file.

It does a good job with Oracle SQL too, not just PL/SQL. sqloracle.vim works differently (not better or worse), and folding support is simpler. You may be satisifed with sql.vim/sqloracle.vim for SQL, especially if you write SQL for multiple databases.

From Supplicant to Maintainer

I tried to find the maintainer of the file to offer my work. Sadly, that individual’s email account is no longer recognized by the provider, and he has not updated anything online in many years.

I contacted the vim-dev mailing list asking what to do. Their suggestion was to assume ownership of the maintainer role. Balking at first, I quickly ran out of excuses I was willing to live with and agreed to do it. Once committed, I felt obligated to go at it full-bore.

I spent way more of my evenings trying to understand the arcane rules of both vim regular expressions and the API rules for syntax, especially region’s, than I care to quantify. There are limitations that made it exceedingly difficult to do folding the way I initially envisioned it.

Vim PL/SQL Syntax File

vim_plsql_syntax repository now lives on github. It contains both the syntax file plsql.vim and the colors file lee.vim. The colors file is completely gratuitous. You don’t need to use it to take advantage of the updates in plsql.vim.

The repository README.md is extensive. It includes screenshots and an installation section.

At this time the main branch contains the version submitted to the vim project for inclusion in version 9 (which should be released soon for some definition of soon). You should select a more current branch, which although they are works in process, are passing my eyeball tests. Right now the branch procedure_folding is stable with some nice new folding features. It is likely to be merged into main soon.

Conclusion

I think I’ve done good work here, but if this is something that interests you, you will have to decide for yourself. I’m pretty happy with it. I will of course attempt to accomodate suggestions. Submitting a Pull Request is the best way to get what you want.

The secondary conclusion is to be careful when you report a problem. It isn’t that uncommon to be told “so fix it.” I’m apparently a really slow learner on that lesson.

Installing Perl DBD::Oracle on RHL

Thu, 28 Apr 2022 12:30:00 -0400

Introduction

Over the years I’ve had many opportunities to struggle with compiling Perl and DBD::Oracle on many different flavors of Unix. It was never easy.

When package management (yum, etc..) became a thing and I eagerly started looking at it, reality slapped me back down. Nobody was packaging DBD::Oracle and the reason seemed to be the Oracle client library license requirements.

When I went looking to do it all again on my home Linux instances, I was pleasantly surprised to find DBD::Oracle was already included on my Oracle Linux Server release 7.9 server. Oracle apparently doesn’t have an issue getting a distribution license for their own dogfood and were able to include DBD::Oracle. Nice!

I also have a Red Hat Enterprise Linux release 8.5 (Ootpa) server running under Microsoft Hyper-V. In spite of the close business relationship Oracle has with Red Hat, RHL does not have DBD::Oracle installed, nor was it found in a dnf search. Note that I’ve already installed Oracle Database on this puppy, so I’ve added Oracle repositories. If there is a yum repository out there that contains DBD::Oracle for Linux, I haven’t found it.

So here we go again.

I know before I start that I need my shell environment to be set so that I can connect with sqlplus. My database SID is rhl1db which is a container database. This works for me given how I installed Oracle:

export ORACLE_HOSTNAME=$(hostname)
export ORAENV_ASK=NO
export ORACLE_SID=rhl1db
. oraenv -s

From the README for DBD::Oracle I know I need to set a variable ORACLE_USERID with the connection string. I could include the @pdbname part there I think, but I also know I can use TWO_TASK to specify the PDB. Remember my SID is for the Container DB. I’m going to connect to a PDB named rhl1pdb.

export TWO_TASK=rhl1pdb
export ORACLE_USERID=lee/MYPASSWORD
sqlplus $ORACLE_USERID
# BINGO! I'm in

At this point I’ve taken care of prerequisites in my environment for doing the make. We could download it and run the steps outlined in the README, but I thought I would swing for the fence.

Trying CPAN

First, I tried to run a cpan shell.

perl -MCPAN -e shell

Yeah, we could be so lucky. It wasn’t there, but ahoy! There is a yum package for it.

I’m running as root. I realize we are supposed to use sudo. You do you, and leave the old dinosaur alone.

dnf search perl-CPAN
# bingo
dnf install perl-CPAN
perl -MCPAN -e shell

Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v2.18)
Enter 'h' for help.

cpan[1]> install DBD::Oracle
Reading '/root/.local/share/.cpan/Metadata'
    Database was generated on Thu, 28 Apr 2022 07:29:03 GMT
Running install for module 'DBD::Oracle'
...

My first attempt the make failed for missing library aio. A little google search and we find we need to install libaio-devel

dnf install libaio-devel
perl -MCPAN -e shell
cpan[1]> install DBD::Oracle
...

This time it compiles, but I run into trouble in the tests. It wants Test::More and Devel::Peek. OK, I’ll try to get those.

perl -MCPAN -e shell
install Test::More
# that worked
install Devel::Peek
# no dice

Devel::Peek is tied up somehow. I didn’t try to figure it out. I know that not all tests will pass anyway. How would you know that? You wouldn’t. It is the source of endless frustration for people new to this. But if you google install issues for DBD::Oracle you will see the blase response to just ignore the failed tests (when there are only a few non-critical failures!) and run make install anyway.

This time when I ran the install from CPAN shell it completed the majority of the tests and I honestly don’t care about the failures.

Scanning cache /root/.local/share/.cpan/build for sizes
...
Files=41, Tests=1867, 17 wallclock secs ( 0.27 usr  0.05 sys +  3.86 cusr  0.68 csys =  4.86 CPU)
Result: FAIL
Failed 5/41 test programs. 2/1867 subtests failed.
make: *** [Makefile:1105: test_dynamic] Error 2
  ZARQUON/DBD-Oracle-1.83.tar.gz
  /usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
  reports ZARQUON/DBD-Oracle-1.83.tar.gz
Failed during this command:
 ZARQUON/DBD-Oracle-1.83.tar.gz               : make_test NO

cpan[2]>

We are left with it not installed though. Bummer. Maybe there is a way to tell the CPAN shell to install even after a failed test, but I didn’t bother. At the top of the block above note that it tells me it is doing this make in a cache directory “/root/.local/share/.cpan/build”. I go down that path and find “DBD-Oracle-1.83-0”. In that directory is the Makefile that was created for this build. I do

make install

All done.

Testing DBD::Oracle

My test script (sans my database password and notice I’m not root anymore):

#!/usr/bin/env perl
use DBI;
use DBD::Oracle;
use strict;
use warnings;

my $dbh = DBI->connect('dbi:Oracle:', 'lee@rhl1pdb', 'my secret password'
                        , {RaiseError => 1, AutoCommit => 0, RowCacheSize => -102400, ora_module_name => 'Perl' }
                      ) or die "Database connection not made: DBI::errstr";
$dbh->do("alter session set nls_date_format = 'mm/dd/yyyy'");
print $dbh->selectrow_array("SELECT sysdate from dual"), "\n";
$dbh->disconnect;

And the output:

lee@rhl1 [/home/lee/git/perl_oracle]
$ ./test_dbd_oracle.pl 
04/28/2022

That may have been the easiest install of DBD::Oracle I’ve ever done.

Granted, I had already installed the Oracle database on this server, configured it, and configured connections (listener, TNS, etc…). I had all that working before I started. If you are on a new machine you must first install the Oracle client and get that all working. That’s also easier than it used to be, but no picnic.

Hope this helped.

Profiling PL/SQL to Examine Context Switch Penalty

Sun, 10 Apr 2022 12:30:00 -0400

Introduction

In a prior post, Cost of UDT Object Methods in SQL, I hypothesized a performance issue was due to Context Switching between SQL and PL/SQL engine. Moving the suspect PL/SQL function call into a PIPELINED Table function solved the issue.

I had three outstanding doubts about what is really going on.

The magnitude of the performance penalty caught me off gaurd. I’ve remediated context switch issues with function calls in a SQL SELECT list before. I haven’t seen one where the impact was this dramatic except where the PL/SQL function was also calling SQL.
I am unsure whether Object type method calls in a SELECT list incur the context switch penalty. I still do not know and this analysis did not answer that question, but it did put the related subject of collection creation/extension cost into perspective.
Unrelated, documentation for chained PIPELINED Table functions suggests bulk collect logic is not needed. I wanted to verify that.

PL/SQL Hierarchical Profiler

If you haven’t used the profiler before, this Jeff Smith post is a nice shortcut for getting started with SqlDeveloper taking care of some of the details. Remember to recompile with debug any of your packages and types that are called by the outer block if you want details for them. Once you get your feet wet, you can read the Oracle document Database Development Guide, Chapter 15, Using the PL/SQL Hierarchical Profiler.

Chained PIEPLINED Table Functions and Bulk Fetch

My working solution used chained PIPELINED Table functions. Per what I perceived the documetation to be implying (without ever coming out and saying it), I implemented the function that reads the cursor from the chain (not the first entry in the chain) without any bulk collection/array processing. The profiler showed that function taking 3.1% of the execution time.

Adding BULK COLLECT LIMIT 100 to the cursor fetch and processing the resulting array caused it to take 3.2% of the execution time (which could be noise) and there was no reduction in total run time. This appears to confirm that there is no advantage to buffering that cursor from one pipe row call to another. It is extra, unneeded processing.

Refactored Code to Analyze

The two variants we are comparing share most code. The one that does almost everything in the PL/SQL engine with a chained pipeline function is called like so:

        v_sql := q'[INSERT /*+ APPEND WITH_PLSQL */ INTO ora$ptt_csv 
WITH
a AS (
    SELECT t.p AS pu
        --perlish_util_udt(t.arr) AS pu
    FROM TABLE(
                app_csv_pkg.split_lines_to_fields(
                    CURSOR(SELECT * 
                           FROM TABLE( app_csv_pkg.split_clob_to_lines(:p_clob, p_skip_lines => 1) )
                    )
                    , p_separator => :p_separator, p_strip_dquote => :p_strip_dquote, p_keep_nulls => 'Y'
                )
    ) t
) SELECT ]'
        -- must use table alias and fully qualify object name with it to be able to call function or get attribute of object
        -- Thus alias x for a and use x.p.get vs a.p.get.
        ||v_cols.map('X.pu.get($##index_val##) AS "$_"').join('
,')
        ||'
FROM a X';
        DBMS_OUTPUT.put_line(v_sql);
        EXECUTE IMMEDIATE v_sql USING  p_clob, p_separator, p_strip_dquote;

The one that does the perlish_util_udt constructor call (which calls app_csv_pkg.split_csv) from SQL is called like so:

        v_sql := q'{INSERT /*+ APPEND WITH_PLSQL */ INTO ora$ptt_csv 
WITH 
a AS (
    SELECT perlish_util_udt(
            p_csv => t.s
            ,p_separator => :p_separator, p_strip_dquote => :p_strip_dquote, p_keep_nulls => 'Y'
            ,p_expected_cnt => :p_expected_cnt
        ) AS pu
    FROM TABLE( app_csv_pkg.split_clob_to_lines(:p_clob, p_skip_lines => 1) ) t
) SELECT }'
        -- must use table alias and fully qualify object name with it to be able to call function or get attribute of object
        -- Thus alias x for a and use x.p.get vs a.p.get.
        ||v_cols.map('X.pu.get($##index_val##) AS "$_"').join('
,')
        ||'
FROM a X';
        DBMS_OUTPUT.put_line(v_sql);
        EXECUTE IMMEDIATE v_sql USING  p_separator, p_strip_dquote, v_cols.arr.COUNT, p_clob ;

The perlish_util_udt constructor calls app_csv_pkg.csv. Both are called directly from PL/SQL in the first variant, but from SQL in the second. That is where we focus for this analysis.

Both were compiled with the following settings for the initial timed run:

ALTER SESSION SET plsql_code_type = NATIVE;
ALTER SESSION SET plsql_optimize_level=3;

Using a 10,766 row, 20 column CLOB input file the total run times are

Variant	Run Time
PL/SQL	9.0
SQL	94.9

If you read the prior article you will notice these times do not foot to those. There were multiple optimizations to the code since that article, most notably using REGEXP_INSTR to parse the CSV rows rather than REGEXP_SUBSTR. There is still a large percentage disparity between the two variants, though both are much faster now.

Running with Hierarchical Profile Enabled

The app_csv_pkg.split_csv procedure is modifed here to support additional data capture. You can see the original full package at https://github.com/lee-lindley/plsql_utilities.

    PROCEDURE split_csv (
         po_arr OUT NOCOPY  &&d_arr_varchar2_udt.
        ,p_s                CLOB
        ,p_separator        VARCHAR2    DEFAULT ','
        ,p_keep_nulls       VARCHAR2    DEFAULT 'N'
        ,p_strip_dquote     VARCHAR2    DEFAULT 'Y' -- also unquotes \" and "" pairs within the field to just "
        ,p_expected_cnt     NUMBER      DEFAULT 0 -- will get an array with at least this many elements
    ) 
    -- when p_s IS NULL, returns initialized collection with COUNT=0
    IS
        v_str                   VARCHAR2(32767);    -- individual parsed values cannot exceed 4000 chars
        v_i                     BINARY_INTEGER := 0;
        v_pos                   BINARY_INTEGER;
        v_pos_last              BINARY_INTEGER := 1;
        v_last_had_separator    BINARY_INTEGER := 0;
        v_len                   BINARY_INTEGER;
        v_regexp                VARCHAR2(1024) := REPLACE(gc_csv_regexp, '__p_separator__', p_separator);

The local procedures defined next are to separate calls to built-in methods so that the profiler can capture the time spent on each separately.

        -- these are to get profile info
        PROCEDURE l_create_arr IS BEGIN po_arr := &&d_arr_varchar2_udt.(); END;
        PROCEDURE l_extend_arr(p_cnt NUMBER) IS BEGIN po_arr.EXTEND(p_cnt); END;
        PROCEDURE l_instr IS BEGIN v_pos := REGEXP_INSTR(p_s, v_regexp, v_pos_last, 1, 1); END;
        PROCEDURE l_substr IS BEGIN v_str := TRIM(SUBSTR(p_s, v_pos_last, v_len)); END;
        PROCEDURE l_repl_sep IS BEGIN v_str := REGEXP_REPLACE(v_str, '\\('||p_separator||')', '\1', 1, 0); END;
        PROCEDURE l_strip_dq IS BEGIN 
	                        v_str := REGEXP_REPLACE(v_str, 
	                                    '^"|"$'         -- leading " or ending "
	                                    ||'|["\\]'  -- or one of chars " or \
	                                        ||'(")'     -- that is followed by a " and we capture that one in \1
	                                    ,'\1'           -- We put any '"' we captured back without the backwack or " quote
	                                    ,1              -- start at position 1 in v_str
	                                    ,0              -- 0 occurence means replace all of these we find
	                                ); 
        END;
	BEGIN
        --po_arr := &&d_arr_varchar2_udt.();
        l_create_arr;
        IF p_expected_cnt > 0 THEN
            --po_arr.EXTEND(p_expected_cnt);
            l_extend_arr(p_expected_cnt);
        END IF;
        IF p_s IS NOT NULL THEN
            LOOP
                --v_pos := REGEXP_INSTR(p_s, v_regexp, v_pos_last, 1, 1); -- get end char of matching string
                l_instr;
                -- the regexp WILL match until it matches the end of the string. Once v_pos_last 
                -- is on a character past the end of the string, it will return 0.
                EXIT WHEN v_pos = 0;
                v_last_had_separator := CASE WHEN SUBSTR(p_s, v_pos - 1, 1) = p_separator THEN 1 ELSE 0 END;
                v_len := (v_pos - v_pos_last) - v_last_had_separator;
                IF v_len > 0 THEN
                    --v_str := TRIM(SUBSTR(p_s, v_pos_last, v_len)); -- could still be null after trim
                    l_substr;
                ELSE
                    v_str := NULL;
                END IF;
                IF v_str IS NOT NULL OR p_keep_nulls = 'Y' THEN
                    IF SUBSTR(v_str,1,1) = '"' THEN
                        IF p_strip_dquote = 'Y' THEN -- otherwise keep everything after trim which means should end on dquote
                            /*
	                        v_str := REGEXP_REPLACE(v_str, 
	                                    '^"|"$'         -- leading " or ending "
	                                    ||'|["\\]'  -- or one of chars " or \
	                                        ||'(")'     -- that is followed by a " and we capture that one in \1
	                                    ,'\1'           -- We put any '"' we captured back without the backwack or " quote
	                                    ,1              -- start at position 1 in v_str
	                                    ,0              -- 0 occurence means replace all of these we find
	                                ); 
                            */
                            l_strip_dq;
                        END IF;
                    ELSIF v_str IS NOT NULL THEN
                        --v_str := REGEXP_REPLACE(v_str, '\\('||p_separator||')', '\1', 1, 0);
                        l_repl_sep;
                    END IF;

                    v_i := v_i + 1;
                    IF v_i > p_expected_cnt THEN -- otherwise we already have room
	                    --po_arr.EXTEND;
                        l_extend_arr(1);
                    END IF;
	                po_arr(v_i) := v_str;
                END IF;
                v_pos_last := v_pos; -- walk the string to next token
            END LOOP;
            IF v_last_had_separator = 1 AND p_keep_nulls = 'Y' THEN -- trailing null
                v_i := v_i + 1;
                IF v_i > p_expected_cnt THEN -- otherwise we already have room
                    --po_arr.EXTEND;
                    l_extend_arr(1);
                END IF;
                po_arr(v_i) := NULL; -- do not think this is necessary, but make it explicit
            END IF;
        END IF; -- end if input string not null
	END split_csv
	;

Recompiling in DEBUG mode and with Profiling turned on, I expected some overhead to add to the run times and we do see that.

ALTER TYPE perlish_util_udt COMPILE DEBUG;
ALTER PACKAGE app_csv_pkg COMPILE DEBUG;
BEGIN
    DBMS_HPROF.start_profiling('PLSHPROF_DIR', 'test.trc');
-- clob not listed for brevity
APP_CSV_PKG.create_ptt_csv(TO_CLOB(q'{...
...
}')
);
    DBMS_HPROF.stop_profiling;
END;
/
SELECT COUNT(*)
FROM ora$ptt_csv
/
COMMIT
/
DECLARE
    runid NUMBER;
BEGIN
    runid := DBMS_HPROF.analyze(LOCATION => 'PLSHPROF_DIR', FILENAME => 'test.trc');
    DBMS_OUTPUT.PUT_LINE('runid = ' || runid);
END;
/

The times are longer with profiling and debug mode, but still proportional.

Variant	Run Time
PL/SQL	13.0
SQL	141.7

Reading the Profiler Data

SqlDeveloper can show it to you including the HTML report it causes to be created in the database directory. That is convenient and I use it, but for this article I teased out the data I wanted to show.

with a as (
select function, namespace, sum(subtree_elapsed_time) as subtree_elapsed_time, sum(function_elapsed_time) as function_elapsed_time
from dbmshp_function_info
where runid = 5
group by function, namespace
), b as (
select function, namespace, sum(subtree_elapsed_time) as subtree_elapsed_time, sum(function_elapsed_time) as function_elapsed_time
from dbmshp_function_info
where runid = 6
group by function, namespace
) select NVL(b.function, a.function) as function
    ,NVL(b.namespace, a.namespace) as namespace
    ,TO_CHAR(a.subtree_elapsed_time/1000000, '999,999.9') AS called_from_sql_cum
    ,TO_CHAR(b.subtree_elapsed_time/1000000, '999,999.9') AS called_from_plql_cum
    ,TO_CHAR(a.function_elapsed_time/1000000, '999,999.9') AS called_from_sql_secs
    ,TO_CHAR(b.function_elapsed_time/1000000, '999,999.9') AS called_from_plsql_secs
from a
full outer join b
    on (b.function = a.function --and b.line# = a.line#
    )
        OR (b.function = 'CREATE_PTT_CSV' AND a.function = 'CREATE_PTT_CSV_UDT')
order by NVL(a.subtree_elapsed_time, b.subtree_elapsed_time) desc
    ;

You may notice that the total times are a bit less than reported above. The difference I believe is that I’m reporting above the elapsed time in sqlplus, which includes the time to load the CLOB into memory.

PL/SQL Hierarchical Profiler Data

I find it interesting that two of the operations I was concerned with, creating the collection (SPLIT_CSV.L_CREATE_ARR) and extending/growing the collection (SPLIT_CSV.L_EXTEND_ARR), are non-factors.

My original premise was that context switching was the villain. If that were the entire story, I would expect the overall time for the function called from SQL to show the longer run time while the internal components of the call did not. In other words the impact of the context switch should only be at the beginning and end of the single call as we move memory around for the context switch. That is NOT what we see here.

Individual operations that should be taking place inside the PL/SQL engine for both runs have an elapsed time sum that is ten to twenty times longer when the function is called from SQL than when called from PL/SQL.

It is possible that I do not understand all implications of “context switching” here.

If the only discrepancies were in the regular expression engine, I would feel much better as that beast could be a weirdo. But we have anomalous behavior in the simple substring operation as well (SPLIT_CSV.L_SUBSTR) and that has nothing to do with the regular expression engine.

What the problem operations all have in common is reading and/or writing character data in memory. It may be that each of these operations incur the context switch penalty as they negotiate the “other” memory space.

I’m now at a point where I need to revisit my understanding of what context switch means for the SQL/PL_SQL interface. It isn’t like I haven’t been reading everything I could find on the subject for the last two weeks, yet somehow, exactly what goes on here has eluded me.

Stay tuned.

Oracle REGEXP_INSTR and Beginning of Line Anchor

Sun, 10 Apr 2022 02:45:00 -0400

The Problem

REGEXP_INSTR, REGEXP_COUNT, REGEXP_REPLACE, and REGEXP_SUBSTR all have a position parameter defined as

position is a positive integer indicating the character of source_char where Oracle should begin the search. The default is 1, meaning that Oracle begins the search at the first character of source_char.

This is handy when you want to walk through a string applying the regular expression starting at different points, such as after the last match.

Problem: When position is not 1, the beginning of line anchor ‘^’ does not match the beginning of the substring.

There is also an occurence parameter that can be used to similar effect. I presume that internally the regular expression engine keeps track of the last match rather than parsing the entire string again, but Oracle does not say. Without details about it I’m leary of trusting it for high performance, and in fact have some tangential evidence that using occurence for this purpose is not as performant as using position. I could be wrong but am not going to try to prove it today.

REGEXP_INSTR

This matches the beginning of the string which is two space characters. Since we specify return_opt=1, we are returned the character position AFTER the matched string. This meets expectations.

SELECT REGEXP_INSTR('  <<= string starts with space', '^\s*'
                    , /*position*/ 1, /*occurence */ 1, /*return_opt*/ 1
                   ) AS end_position
FROM dual;

Same thing but without any leading spaces in source_char. Since we have the zero or more modifer for spaces, we still expect a match on the zero width start of line anchor.

SELECT REGEXP_INSTR('String starts with S', '^\s*'
                    , /*position*/ 1, /*occurence */ 1, /*return_opt*/ 1
                   ) AS end_position
FROM dual;

We do get a match (non-zero return value). As expected the character position AFTER the matched string (which is 0 length) is still 1.

What happens when we advance the position so that it is no longer at the start of source_char? Here I set position to 2 so that we start looking for a match at character position 2.

SELECT REGEXP_INSTR('String starts with S'
                    , '^\s*', /*position*/ 2, /*occurence */ 1, /*return_opt*/ 1
                   ) AS end_position
FROM dual;

That was a huge surprise to me. The ‘^’ anchor no longer matches the beginning of what we think of as the string we are matching (sub-string starting at position 2). I don’t know that I can say it is a bug because Oracle does not explain how position is applied, and Oracle is careful in the wording that ‘^’ matches the start of the entire source string (or after a newline when the ‘m’ match_param is specified). Nowhere does it say that ‘^’ matches at starting position.

Needless to say, it does NOT work the way I expect which is as demonstrated by using SUBSTR to achieve the goal of starting at position 2 rather than using the position argument to REGEXP_INSTR.

SELECT REGEXP_INSTR(SUBSTR('String starts with S',2)
                    , '^\s*', /*position*/ 1, /*occurence */ 1, /*return_opt*/ 1
                   ) AS end_position
FROM dual;

We can work with this if we must, but we are making a copy of the rest of the string to do it rather than walking through it in place. This does not make me happy if I’m writing a tight subroutine that is called millions of times.

If we can write the pattern such that it works correctly without ‘^’, we can be efficient using position.

REGEXP_COUNT

The behavior is the same for REGEXP_COUNT as REGEXP_SUBSTR. Starting with the *position=1 we get what we expect.

SELECT REGEXP_COUNT('XXX
Xab
Xcd'
                    ,'^X'
                    , /*position*/ 1, /*match_param*/ 'm'
                   ) AS cnt
FROM dual;

Moving to position=2 my expectation would be to match the ‘X’ in the second character position along with the ones after newlines to give an answer of 3.

SELECT REGEXP_COUNT('XXX
Xab
Xcd'
                    ,'^X'
                    , /*position*/ 2, /*match_param*/ 'm'
                   ) AS cnt
FROM dual;

Of course the behavior is same as for REGEXP_INSTR and we do not match until after a newline.

Interestingly if we advance to position=4 which means it is sitting on a newline character as the starting position for the string, we still get an answer of 2.

SELECT REGEXP_COUNT('XXX
Xab
Xcd'
                    ,'^X'
                    , /*position*/ 4, /*match_param*/ 'm'
                   ) AS cnt
FROM dual;

When we advance to character position 5 which is after the newline, the ‘^’ no longer matches at our start of string position and the first X on line 2 is not matched. We only match on the 3rd line.

SELECT REGEXP_COUNT('XXX
Xab
Xcd'
                    ,'^X'
                    , /*position*/ 5, /*match_param*/ 'm'
                   ) AS cnt
FROM dual;

REGEXP_REPLACE

Starting position of 1 works as expected as we anchor at the beginning of source_char. The starting ‘X’ on all three lines is replaced by ‘Y’.

SELECT REGEXP_REPLACE('XXX
Xab
Xcd'
                    ,'^X', 'Y'
                    , /*position*/ 1, /*occurence*/ 0, /*match_param*/ 'm'
                   ) AS x_to_y
FROM dual

YXX
Yab
Ycd

Starting position of 2 my expectation is that the ‘X’ in character position 2 of line 1 be changed to ‘Y’. That does not happen.

SELECT REGEXP_REPLACE('XXX
Xab
Xcd'
                    ,'^X', 'Y'
                    , /*position*/ 2, /*occurence*/ 0, /*match_param*/ 'm'
                   ) AS x_to_y
FROM dual;

XXX
Yab
Ycd

REGEXP_SUBSTR

Starting position of 1 works as expected.

SELECT REGEXP_SUBSTR('XXX
Xab
Xcd'
                    ,'^X.*'
                    , /*position*/ 1, /*occurence*/ 1, /*match_param*/ 'm'
                   ) AS x_to_y
FROM dual;

XXX

Setting occurence to 2 also works as expected.

SELECT REGEXP_SUBSTR('XXX
Xab
Xcd'
                    ,'^X.*'
                    , /*position*/ 1, /*occurence*/ 2, /*match_param*/ 'm'
                   ) AS x_to_y
FROM dual;

Xab

Setting position to 2 we do not get a match until after a newline which is same behavior as the other regular expression functions.

SELECT REGEXP_SUBSTR('XXX
Xab
Xcd'
                    ,'^X.*'
                    , /*position*/ 2, /*occurence*/ 1, /*match_param*/ 'm'
                   ) AS x_to_y
FROM dual;

Xab

Conclusion

Maybe Oracle can claim this is working as designed. As far as I’m concerned it is a bug, but not one I expect them to fix. It would break too much code already dependent on this behavior. Oracle should amend the documentation to explain this behavior as it is different from the way we work with regular expressions to walk through a string in other languages like Perl.

Cost of UDT Object Methods in SQL

Sat, 02 Apr 2022 03:30:00 -0400

Introduction

In a prior post, CSV Clob and Private Temporary Table, I described how one could use some new tools I created to parse a CSV clob into records and fields, then use that parsed data in a SQL statement, perhaps including DML.

The article was about a work in progress and the code has changed, but it will give you an idea of how we got here if you are interested.

I ran into performance issues for large datasets while testing it. This article is about the techniques I tried for tuning it, and a fact about Oracle Object Type methods that I had not thought about much before.

Note: you will see what may be unfamiliar methods of perlish_util_udt Object type in this code. They work like Perl operators on lists.

SQL Engine and PL/SQL Method Calls

Here is an example query that is built as dynamic SQL in a PL/SQL procedure. It returns a resultset built from reading CSV data provided to the program as a CLOB.

You can find the packages at https://github.com/lee-lindley/plsql_utilities. The procedure I’m showing examples from is app_csv_pkg.create_ptt_csv.

WITH a AS (
    SELECT perlish_util_udt(
            app_csv_pkg.split_csv(t.s, p_separator => :p_separator, p_strip_dquote => :p_strip_dquote, p_keep_nulls => 'Y')
        ) AS p
    FROM TABLE( app_csv_pkg.split_clob_to_lines(:p_clob, p_skip_lines => 1) ) t
) SELECT 
    x.p.get(1) AS "Employee ID"
    ,x.p.get(2) AS "Last Name"
    ,x.p.get(3) AS "First Name"
    ,x.p.get(4) AS "nickname"
FROM a x

The pipelined table function split_clob_to_lines is efficient. What turned out to be not efficient is the call to the perlish_util_udt object conststructor and/or the call to the function split_csv in the SQL SELECT list. Those are both efficient when called inside a PL/SQL program.

This should not have surprised me. I know that the SQL engine must do a context switch for each call to a PL/SQL program unless it is cached. In this case the cost was much, much larger than I expected.

For a 10K row CLOB of CSV data (20 fields each row) the above construct ran for 20 minutes to process the query on an AIX based database, and 6 minutes on my Intel NUC running in a Hyper-V Linux partition. It is difficult to tell what it is doing from the WAIT events, but I suspected it was in-memory operations and context switching.

Removing the Object

I also suspected the object method get calls might be doing context switches. My first refinement attempt was to remove the object from the picture and use a WITH FUNCTION to snag the field values from the array returned by split_csv.

WITH FUNCTION wget(
    p_arr   ARR_VARCHAR2_UDT
    ,p_i    NUMBER
) RETURN VARCHAR2
AS
BEGIN
    RETURN p_arr(p_i);
END;
a AS (
    SELECT app_csv_pkg.split_csv(t.s
                    , p_separator => :p_separator, p_strip_dquote => :p_strip_dquote, p_keep_nulls => 'Y'
            ) AS arr
    FROM TABLE( app_csv_pkg.split_clob_to_lines(:p_clob, p_skip_lines => 1) ) t
) SELECT 
     get(a.arr, 1) AS "Employee ID"
    ,get(a.arr, 2) AS "Last Name"
    ,get(a.arr, 3) AS "First Name"
    ,get(a.arr, 4) AS "nickname"
FROM a x

This also ran for 6 minutes on my Intel NUC database. We haven’t ruled out that the object could play a part, but we know that the function call split_csv is a problem.

If you are thinking about using a UDF pragma on that puppy (split_csv), I could not get any of the variations I tried to give an improvement. There are many examples shown of the limitations of that pragma (or using a WITH function) that does anything very complicated.

Make the Function Pipelined in a Chain

The next refactoring I tried was moving the call to split_csv into another Pipelined Table Function to be called in a chain with split_clob_to_lines. I called it split_lines_to_fields (catcalls from the peanut gallery may be deserved). Maybe I’ll come up with a better name before I merge this code.

From the package header:

    TYPE t_csv_fields_rec IS RECORD(
        arr ARR_VARCHAR2_UDT
        ,rn NUMBER
    );
    TYPE t_arr_csv_fields_rec IS TABLE OF t_csv_fields_rec;

    FUNCTION split_lines_to_fields(
        p_curs          t_curs_csv_row_rec
        ,p_separator    VARCHAR2    DEFAULT ','
        ,p_strip_dquote VARCHAR2    DEFAULT 'Y' -- also unquotes \" and "" pairs within the field to just "
        ,p_keep_nulls   VARCHAR2    DEFAULT 'Y'
    ) 
    RETURN t_arr_csv_fields_rec
    PIPELINED
    ;

and from the package body:

    FUNCTION split_lines_to_fields(
        p_curs          t_curs_csv_row_rec
        ,p_separator    VARCHAR2    DEFAULT ','
        ,p_strip_dquote VARCHAR2    DEFAULT 'Y' -- also unquotes \" and "" pairs within the field to just "
        ,p_keep_nulls   VARCHAR2    DEFAULT 'Y'
    ) 
    RETURN t_arr_csv_fields_rec
    PIPELINED
    IS
        v_row       t_csv_fields_rec;
        v_in_row    t_csv_row_rec;
    BEGIN
        LOOP
            FETCH p_curs INTO v_in_row;
            EXIT WHEN p_curs%NOTFOUND;
            v_row.rn := v_in_row.rn;
            v_row.arr := split_csv(v_in_row.s
                                    , p_separator       => p_separator
                                    , p_strip_dquote    => p_strip_dquote
                                    , p_keep_nulls      => p_keep_nulls
            );
            PIPE ROW(v_row);
        END LOOP;
        RETURN;
    END split_lines_to_fields
    ;

The documentation never shows a Bulk Collect in the chained pipeline examples, and the descriptions it uses and other hints make me believe it is not needed. I wish it was more explicit.

As recommended in the documentation, it is called in a chain of pipelined table functions directly in SQL with CURSOR casts as

WITH FUNCTION wget(
    p_arr   ARR_VARCHAR2_UDT
    ,p_i    NUMBER
) RETURN VARCHAR2
AS
BEGIN
    RETURN p_arr(p_i);
END;
a AS (
    SELECT t.arr 
    FROM TABLE(
                app_csv_pkg.split_lines_to_fields(
                    CURSOR(SELECT * 
                           FROM TABLE( app_csv_pkg.split_clob_to_lines(:p_clob, p_skip_lines => 1) )
                    )
                    , p_separator => :p_separator, p_strip_dquote => :p_strip_dquote, p_keep_nulls => 'Y'
                )
    ) t
) SELECT 
     get(a.arr, 1) AS "Employee ID"
    ,get(a.arr, 2) AS "Last Name"
    ,get(a.arr, 3) AS "First Name"
    ,get(a.arr, 4) AS "nickname"
FROM a

This ran in 23 seconds. Moving the function call into a pipelined table chain solves the problem, but I still want to know whether or how much the Object plays a part.

Call the Object Constructor in SQL

I could have moved the object constructor into the pipeline chain (and did try that), but it turns out to not be necessary.

WITH a AS (
    SELECT perlish_util_udt(t.arr) AS p
    FROM TABLE(
                app_csv_pkg.split_lines_to_fields(
                    CURSOR(SELECT * 
                           FROM TABLE( app_csv_pkg.split_clob_to_lines(:p_clob, p_skip_lines => 1) )
                    )
                    , p_separator => :p_separator, p_strip_dquote => :p_strip_dquote, p_keep_nulls => 'Y'
                )
    ) t
) SELECT 
     X.p.get(1) AS "Employee ID"
    ,X.p.get(2) AS "Last Name"
    ,X.p.get(3) AS "First Name"
    ,X.p.get(4) AS "nickname"
FROM a X

This also ran in 23 seconds. I prefer the object way better than the WITH FUNCTION way, but if you don’t already have an object wrapped around your array with a handy method, the WITH FUNCTION is just dandy.

Conclusion

Calls to a PL/SQL function in a SELECT list can lead to significant cost via context switching. This is well known, and there are several caching strategies (DETERMINISTIC, RESULT_CACHE, Scalar Subquery) to mitigate it. Using a Pipelined Table Function chain as I did here is also a viable mitigation strategy, and I believe the appropriate one for this use case.

My concern that object construction and simple method function calls in the SQL engine could context switch seems to be unfounded. I re-read the Object Relational Developer Guide and searched Google several ways looking for more information about how object methods are implemented, but it is not clearly stated. They seem to be neither fish nor fowl, SQL engine nor PL/SQL engine.

I am wondering why using an object type is not touted as another mitigation strategy for PL/SQL Function context switching (at least for when the method does not need to call anything other than builtin functions)? I realize we have WITH FUNCTIONs, UDF pragma, and with Oracle 21c, SQL Macros, but it would be nice to know if Object methods are an alternative.

I may explore this as a context switching mitigation strategy another day.

Appendix - The Full Monty

If you have a use case that calls for the ultimate in performance (this one does not, but I happened to try this early in my analysis), you can get down and dirty with DBMS_SQL and I believe what is called “Method 4” dynamic SQL. That classification is because we have a variable number of bind elements.

Unlike the other examples, this one shows the entire procedure that creates a Private Temporary Table (PTT) and populates it from the CSV Clob.

The best I could achieve with the code shown in the article was 23 seconds. This version operates in 19 seconds. It isn’t worth the complexity for this package, but it is nice to know how to do it for the rare occassions when you might need to squeeze out that last little bit of efficiency.

The first part is mostly the same as the code I didn’t show you for the other variations. It parses the first row of the CLOB to get the column names and creates the PTT.

    PROCEDURE create_ptt_csv (
         --
         -- creates private temporary table "ora$ptt_csv" with columns named in first row of data (case preserved).
         -- from a CLOB containing CSV lines.
         -- All fields are varchar2(4000)
         --
	     p_clob         CLOB
	    ,p_separator    VARCHAR2    DEFAULT ','
	    ,p_strip_dquote VARCHAR2    DEFAULT 'Y' -- also unquotes \" and "" pairs within the field to just "
    ) IS
        v_cols          perlish_util_udt; -- for manipulating column names into SQL statement
        v_sql           CLOB;
        v_first_row     VARCHAR2(32767);
        v_ins_curs      INT;
        v_num_rows      INT;
        v_last_row_cnt  BINARY_INTEGER := 0;
        v_col_cnt       BINARY_INTEGER;
        v_vals_1_row    ARR_VARCHAR2_UDT;  -- from split_csv on 1 line
        v_rows          DBMS_SQL.varchar2a;     -- from split_clob_to_lines fetch
        --
        -- variable number of columns, each of which has a bind array.
        --
        TYPE varchar2a_tab  IS TABLE OF DBMS_SQL.varchar2a INDEX BY BINARY_INTEGER;
        v_vals          varchar2a_tab;          -- array of columns each of which holds array of values
        --
        -- We get all but the header row when we read the clob in a loop.
        --
        CURSOR c_read_rows IS
            SELECT t.s
            FROM TABLE(app_csv_pkg.split_clob_to_lines(p_clob, p_skip_lines => 1))  t
            ;
    BEGIN
        BEGIN
            -- read the first row only
            SELECT s INTO v_first_row 
            FROM TABLE( app_csv_pkg.split_clob_to_lines(p_clob, p_max_lines => 1) )
            ;
            IF v_first_row IS NULL THEN
                raise_application_error(-20222,'app_csv_pkg.create_ptt_csv did not find csv rows in input clob.');
            END IF;
        EXCEPTION WHEN NO_DATA_FOUND THEN
            raise_application_error(-20222,'app_csv_pkg.create_ptt_csv did not find csv rows in input clob.');
        END;
        -- split the column header values into collection
        v_cols := perlish_util_udt(split_csv(v_first_row, p_separator => p_separator, p_strip_dquote => 'Y'));
        v_col_cnt := v_cols.arr.COUNT;

        -- create the private global temporary table with "known" name and columns matching names found
        -- in csv first record
        --
        v_sql := 'DROP TABLE ora$ptt_csv';
        BEGIN
            EXECUTE IMMEDIATE v_sql;
        EXCEPTION WHEN OTHERS THEN NULL;
        END;

        v_sql := 'CREATE PRIVATE TEMPORARY TABLE ora$ptt_csv(
'
            ||v_cols.map('"$_"    VARCHAR2(4000)').join('
,')
            ||'
)'
            ;
        DBMS_OUTPUT.put_line(v_sql);
        EXECUTE IMMEDIATE v_sql;

The next part populates the PTT. This variation uses bulk collect to read the data as lines, splits the lines into columns, stuffs the column data into individual arrays, then executes a DBMS_SQL INSERT cursor bound to those arrays. Pretty slick, if a little hard core.

        -- 
        -- Dynamic sql for dbms_sql. will be used with bind arrays.
        -- Of note is that it reports conventional load even if specify append.
        -- I don't understand that as I've seen other reports that direct path load works.
        -- Does not seem to matter though.
        --
        v_sql := 'INSERT INTO ora$ptt_csv(
'
        ||v_cols.map('"$_"').join(', ')             -- the column names in dquotes
        ||'
) VALUES (
'
        ||v_cols.map(':$##index_val##').join(', ')  -- :1, :2, :3, etc... bind placeholders
        ||'
)';
        DBMS_OUTPUT.put_line(v_sql);
        v_ins_curs := DBMS_SQL.open_cursor;
        DBMS_SQL.parse(v_ins_curs, v_sql, DBMS_SQL.native);

        OPEN c_read_rows;
        LOOP
            FETCH c_read_rows BULK COLLECT INTO v_rows LIMIT 100;
            EXIT WHEN v_rows.COUNT = 0;
            FOR i IN 1..v_rows.COUNT
            LOOP
                v_vals_1_row := app_csv_pkg.split_csv(v_rows(i), p_separator => p_separator, p_strip_dquote => p_strip_dquote, p_keep_nulls => 'Y');
                -- j is column number
                FOR j IN 1..v_col_cnt
                LOOP
                    v_vals(j)(i) := v_vals_1_row(j);
                END LOOP;
            END LOOP;

            IF v_last_row_cnt != v_rows.COUNT THEN -- will be true on first loop iteration and maybe last
                v_last_row_cnt := v_rows.COUNT;
                -- bind each column array. v_vals has an array for every column
                FOR j IN 1..v_col_cnt
                LOOP
                    DBMS_SQL.bind_array(v_ins_curs, ':'||TO_CHAR(j), v_vals(j), 1, v_last_row_cnt);
                END LOOP;
            END IF;

            v_num_rows := DBMS_SQL.execute(v_ins_curs);

        END LOOP;
        DBMS_SQL.close_cursor(v_ins_curs);
        CLOSE c_read_rows;
    END create_ptt_csv
    ;

Devops Data Deployment for Oracle

Mon, 28 Mar 2022 10:00:00 -0400

Introduction

Most large corporations have adopted some form of Continuous Improvement (CI) strategy with a Devops build and deploy component. The people who manage and direct these operations tend to be Web guys and the tools tend to focus on deploying Web applications (this includes middleware component on Java and .NET servers). If I am guilty of stereotyping incorrectly, please tell me about organizations with a broader focus. I could be wrong, but that is what I have observed so far.

Other application types are supported, just not as first class citizens. Specifically, database application deployment is a red-headed step-child of Devops. I’m not blaming the Devops guys either. We database practioners are likely guilty of obstruction. Any tool other than what we know works will be looked upon with suspicion.

How Can We Deploy?

We need to deploy database application code (packages, table definitions, etc..) and we need to deploy data (configuration table entries for example).

We have a suite of client tools we can use as Oracle clients for deploying code and data manually, all of which require providing or storing a password:

SQL*PLUS
imp
sqlldr
external tables (assuming you can deliver files to a database directory)
sqlCL
sqlDeveloper
Toad

We can rule out the gui tools SqlDeveloper and Toad. These are not suitable for automated deployment.

SQL*PLUS is perfectly adequate for deploying our code to the database. Oracle uses it to deploy database code and patches. We have all been using SQL*PLUS for as long as we have used Oracle. It is clunky, limiting and antiquated, but it works and we know how to make it do the needful.

Strangely, sqlCL does not seem well supported. I know some reasons for that including security concerns with having Java client software on servers, but the primary reason is likely inertia. This is a shame because sqlCl supports CSV file loading, which would address one of our two deployment needs.

I have not seen a Devops deployment configuration that uses anything other than SQL*PLUS to deploy code. If there are some, my guess is the database application guys are less than thrilled. I could be wrong.

Deploying Data

For batch operations we have mechanisms for transfering files between production systems and/or with outside vendors. We have custom processes using sqlldr, External tables and perhaps an ETL tool to import the data from those files. Yet the mechanisms we have for slinging files around in a production environment are probably not supported through Devops.

We have files in our build that are on the deployment server and we have a SQL*PLUS client.

Traditionally when faced with this we resort to what I call “stupid data load.” We create a series of Insert or Merge statements with hard coded data and run those in SQL*PLUS. Maybe we get a little more ambitious and create a single INSERT or MERGE statement with a series of SELECT FROM DUAL UNION ALL … statements providing the input. It is still “stupid data load.” Works fine for a small number of records. Not so fine when you need to create several thousand records.

If Devops configured a capability to use sqlldr, we could do what we must. The use case for it is limited though and the firm may not choose to invest the resources necessary to configure and manage that capability.

An Alternative for Loading Data with SQL*PLUS

Using tools provided in plsql_utilities we can build deployment scripts that run in SQL*PLUS handling relatively large amounts of data as Comma Separated Value (CSV) records efficiently. Not as efficiently as using sqlldr, but good enough.

From the documentation at plsql_utilities/app_csv_pkg…

Read the data from a table (with an optional WHERE clause) and convert it into a CSV CLOB with a header row. Break the CLOB into a set of quoted string literals. Generate a script that

Creates a Private Temporary Table from the CLOB (input as contactenated string literals).
Inserts or Merges records from the PTT into the target Table

We call the deployment generator:

SELECT APP_CSV_PKG.gen_deploy_merge(
    p_table_name    => 'MY_TABLE_NAME'
    ,p_key_cols     => 'ID'
    ,p_where_clause => 'id <= 10'
) FROM DUAL;

That provides the following script as a CLOB in your query result. Note that if the size of the CSV CLOB holding the records was greater than 32767, it would be represented by a concatenated set of quoted literals instead of just one as shown here.

BEGIN
    APP_CSV_PKG.create_ptt_csv(TO_CLOB(q'{"ID","MSG","DT"
1,"testing...","03/26/2022"
2,"testing...","03/27/2022"
3,"testing...","03/28/2022"
4,"testing...","03/29/2022"
5,"testing...","03/30/2022"
6,"testing...","03/31/2022"
7,"testing...","04/01/2022"
8,"testing...","04/02/2022"
9,"testing...","04/03/2022"
10,"testing...","04/04/2022"
}')
);
END;
/
MERGE INTO MY_TABLE_NAME t
USING (
    SELECT *
    FROM ora$ptt_csv
) q
ON (
    t."ID" = q."ID"
)
WHEN MATCHED THEN UPDATE SET
    t."MSG" = q."MSG"
    ,t."DT" = q."DT"
WHEN NOT MATCHED THEN INSERT(
    "ID", "MSG", "DT"
) VALUES (
    q."ID", q."MSG", q."DT"
);
COMMIT;

Conclusion

I explored multiple mechanisms to avoid “stupid data load.” The obvious best choice would be if the right tools for the job like sqlldr, imp, or sqlCL were available, but we play the hand we are dealt. If you are stuck with SQL*PLUS, this package can be a solution.

A Ruby/Rouge Lexer Class for Oracle PL/SQL

Sun, 20 Mar 2022 12:30:00 -0400

Introduction

This blog is hosted by Github Pages. The standard way to generate the site is via Jekyll which can parse Markdown files. It supports code syntax highlighting via a Ruby gem named Rouge. It is every bit as confusing as it sounds, but strangely familiar if you have been hacking stuff like this for a while.

The nice thing about it is that I was already using Markdown to build my pages and converting them to HTML with pandoc. It was messy and cumbersome, so switching to Github Pages seemed like a good idea. As with anything to do with HTML, CSS and web page development, nothing is ever easy.

Github has decent support for SQL syntax highlighting if you use the fence construct:

```sql
SELECT ...
```

It renders my repository README.md files fairly well. Unfortunately, it uses a different process than Github Pages via Rouge. The SQL lexer for Rouge is not as robust as I wanted, and when I started looking closer, I realize the existing one for Github wasn’t great either.

I looked into what I could do about it.

Rouge

I got a little discouraged at first because I had not picked up Ruby before and wasn’t really interested in learning a new language. Nevertheless I dug into it a bit, got Ruby, Jekyll and Rouge installed locally on a Linux image (apparently it is harder to do on Windows), and started trying to figure out how it fit together.

I can’t tell you how I wound up here, but the documentation for Rouge is very good. I found Lexer Development. I looked at the existing SQL lexer and also the ones for Perl and Ruby.

The code is a state machine that walks through the text using regular expressions and classifying the parsed tokens with names that correspond to the CSS file we use for syntax highlighting it. Even though I had never looked at the language before, it was fairly easy to read and understand what it was doing.

A PL/SQL Lexer

Long story long enough already, I faked it and wound up making what I believe is a decent lexer for Oracle PL/SQL. I did a lot of grunt work looking through the manuals and v$ tables gathering all of the Oracle keywords, reserved words, types and built-in functions which become part of the lexer. The regular expressions for breaking it into tokens is where I think I did a decent job based on my own understanding of how Oracle parses it and similar work I’ve done before. The q-quote operator was a nice challenge that consumed several hours.

I submitted a Pull Request to the Rouge team. There are quite a few in their queue, so I have no idea whether it will fly any time soon, but I followed the guidelines they provided, so we will see.

Sample Output

SELECT e.first_name AS "First Name", e.last_name AS "Last Name"
    , d.department_name AS "Department 
Name"
FROM hr.employees e
JOIN hr.departments d
    ON e.department_id = d.department_id
WHERE e.first_name = 'Bruce' AND e.last_name = 'Lee'
ORDER BY d.department_name
;

CREATE OR 
 REPLACE 
  PACKAGE 
   sample_pkg
    PROCEDURE transform_perl_regexp(p_re VARCHAR2)
    RETURN VARCHAR2
    DETERMINISTIC
    ;
END sample_pkg;
/
CREATE OR REPLACE PACKAGE BODY sample_pkg

    FUNCTION transform_perl_regexp(p_re VARCHAR2)
    RETURN VARCHAR2
    DETERMINISTIC
    IS
        /*
            strip comment blocks that start with at least one blank, then
            '--' or '#', then everything to end of line or string
 ===> shows /* does not start a new comment
        */
 -- quoted string with embedded newline
        c_strip_comments_regexp CONSTANT VARCHAR2(32767) := q'+[[:blank:]](--|#).*($|
)+';
$IF $$is_dummy_needed $then
        c_dummy                 CONSTANT VARCHAR2(30) := 'newline>
<anotherline';
        v_rec hr.employees%ROWTYPE;
        v_dummy2 hr.employees.employee_id%type;
$End
        
    BEGIN
    -- note that \n, \r and \t will be replaced if not preceded by a \
    -- \\n and \\t will not be replaced. Unfortunately, neither will \\\n or \\\t.
    -- If you need \\\n, use \\ \n since the space will be removed.
    -- We are not parsing into tokens, so this is as close as we can get cheaply
      RETURN
        REGEXP_REPLACE(
          REGEXP_REPLACE(
            REGEXP_REPLACE(
              REGEXP_REPLACE(
                REGEXP_REPLACE(p_re, c_strip_comments_regexp, NULL, 1, 0, 'm') -- strip comments
                , '\s+', NULL, 1, 0                 -- strip spaces and newlines too like 'x' modifier
              )
              , q'[(^|[^\\])\\t]', '\1'||CHR(9), 1, 0    -- replace \t with tab character value so it works like in perl
            )
            , q'{(^|[^\\])\\n}', '\1'||CHR(10), 1, 0       -- replace \n with newline character value so it works like in perl
          )
          , q'((^|[^\\])\\r)', '\1'||CHR(13), 1, 0         -- replace \r with CR character value so it works like in perl
        )
      ;

    END transform_perl_regexp;
END  sample_pkg;
/
 -- shows a comment on last line

The Lexer Code

The huge list of keywords makes this unwieldly, but the actual code part is not that big.

# -*- coding: utf-8 -*- #
# frozen_string_literal: true

module Rouge
  module Lexers
    class PLSQL < RegexLexer
      title "PLSQL"
      desc "Procedural Language Structured Query Language for Oracle relational databases"
      tag 'plsql'
      filenames '*.pls', '*.typ', '*.tps', '*.tpb', '*.pks', '*.pkb', '*.pkg', '*.trg'
      mimetypes 'text/x-plsql'

      def self.keywords
        @keywords ||= Set.new(%w(
        ACCESSIBLE AGENT ALL ALTER AND ANY AS ASC BETWEEN BFILE_BASE BLOB_BASE BY
        C CALLING CHARSET CHARSETFORM CHARSETID CHAR_BASE CHECK CLOB_BASE CLUSTER
        COLLATE COMPILED COMPRESS CONNECT CONNECT_BY_ROOT CONSTRUCTOR CREATE CUSTOMDATUM
        DATE_BASE DEFAULT DELETE DESC DISTINCT DROP DURATION ELSE ELSIF EXCEPT EXCLUSIVE
        EXISTS EXIT FIXED FOR FORALL FROM GENERAL GRANT GROUP HAVING IDENTIFIED IN INDEX
        INDICES INSERT INTERFACE INTERSECT INTO IS LARGE LIKE LIMITED LOCK LOOP MAXLEN
        MINUS MODE NOCOMPRESS NOT NOWAIT NULL NUMBER_BASE OCICOLL OCIDATE OCIDATETIME
        OCIDURATION OCIINTERVAL OCILOBLOCATOR OCINUMBER OCIRAW OCIREF OCIREFCURSOR
        OCIROWID OCISTRING OCITYPE OF ON OPTION OR ORACLE ORADATA ORDER ORLANY ORLVARY
        OUT OVERRIDING PARALLEL_ENABLE PARAMETER PASCAL PCTFREE PIPE PIPELINED POLYMORPHIC
        PRAGMA PRIOR PUBLIC RAISE RECORD RELIES_ON REM RENAME RESOURCE RESULT REVOKE ROWID 
        SB1 SB2 SELECT SEPARATE SET SHARE SHORT SIZE SIZE_T SPARSE SQLCODE SQLDATA
        SQLNAME SQLSTATE STANDARD START STORED STRUCT STYLE SYNONYM TABLE TDO THEN
        TRANSACTIONAL TRIGGER UB1 UB4 UNION UNIQUE UNSIGNED UNTRUSTED UPDATE VALIST
        VALUES VARIABLE VIEW VOID WHERE WHILE WITH
        ))
      end
      def self.keywords_nresvd
        @keywords_nresvd ||= Set.new(%w(
        ABORT ABS ABSENT ACCESS ACCESSED ACCOUNT ACL ACOS ACROSS ACTION ACTIONS
        ACTIVATE ACTIVE ACTIVE_COMPONENT ACTIVE_DATA ACTIVE_FUNCTION ACTIVE_TAG ACTIVITY
        ADAPTIVE_PLAN ADD ADD_COLUMN ADD_GROUP ADD_MONTHS ADG_REDIRECT_DML ADG_REDIRECT_PLSQL
        ADJ_DATE ADMIN ADMINISTER ADMINISTRATOR ADVANCED ADVISE ADVISOR AFD_DISKSTRING
        AFFINITY AFTER AGGREGATE AGGREGATES ALGORITHM ALIAS ALLOCATE ALLOW ALL_ROWS
        ALTERNATE ALWAYS ANALYTIC ANALYTIC_VIEW_SQL ANALYZE ANCESTOR ANCILLARY AND_EQUAL
        ANOMALY ANSI_REARCH ANSWER_QUERY_USING_STATS ANTIJOIN ANYSCHEMA ANY_VALUE
        APPEND APPENDCHILDXML APPEND_VALUES APPLICATION APPLY APPROX_COUNT APPROX_COUNT_DISTINCT
        APPROX_COUNT_DISTINCT_AGG APPROX_COUNT_DISTINCT_DETAIL APPROX_MEDIAN APPROX_PERCENTILE
        APPROX_PERCENTILE_AGG APPROX_PERCENTILE_DETAIL APPROX_RANK APPROX_SUM ARCHIVAL
        ARCHIVE ARCHIVED ARCHIVELOG ARRAY ARRAYS ASCII ASCIISTR ASIN ASIS ASSEMBLY
        ASSIGN ASSOCIATE ASYNC ASYNCHRONOUS AS_JSON AT ATAN ATAN2 ATTRIBUTE ATTRIBUTES
        AUDIT AUTHENTICATED AUTHENTICATION AUTHID AUTHORIZATION AUTO AUTOALLOCATE
        AUTOEXTEND AUTOMATIC AUTO_LOGIN AUTO_REOPTIMIZE AVAILABILITY AVCACHE_OP AVERAGE_RANK
        AVG AVMDX_OP AVRO AV_AGGREGATE AV_CACHE AW BACKGROUND BACKINGFILE BACKUP BAND_JOIN
        BASIC BASICFILE BATCH BATCHSIZE BATCH_TABLE_ACCESS_BY_ROWID BECOME BEFORE
        BEGIN BEGINNING BEGIN_OUTLINE_DATA BEHALF BEQUEATH BFILENAME BIGFILE BINARY
        BINARY_DOUBLE_INFINITY BINARY_DOUBLE_NAN BINARY_FLOAT_INFINITY BINARY_FLOAT_NAN
        BINDING BIND_AWARE BIN_TO_NUM BITAND BITMAP BITMAPS BITMAP_AND BITMAP_BIT_POSITION
        BITMAP_BUCKET_NUMBER BITMAP_CONSTRUCT_AGG BITMAP_COUNT BITMAP_OR_AGG BITMAP_TREE
        BITOR BITS BITXOR BIT_AND_AGG BIT_OR_AGG BIT_XOR_AGG BLOCK BLOCKCHAIN BLOCKING
        BLOCKS BLOCKSIZE BLOCK_RANGE BODY BOOL BOOTSTRAP BOTH BOUND BRANCH BREADTH
        BROADCAST BSON BUFFER BUFFER_CACHE BUFFER_POOL BUILD BULK BUSHY_JOIN BYPASS_RECURSIVE_CHECK
        BYPASS_UJVC CACHE CACHE_CB CACHE_INSTANCES CACHE_TEMP_TABLE CACHING CALCULATED
        CALL CALLBACK CANCEL CAPACITY CAPTION CAPTURE CARDINALITY CASCADE CASE CAST
        CATALOG_DBLINK CATEGORY CDB$DEFAULT CDB_HTTP_HANDLER CEIL CELLMEMORY CELL_FLASH_CACHE
        CERTIFICATE CFILE CHAINED CHANGE CHANGE_DUPKEY_ERROR_INDEX CHARTOROWID CHAR_CS
        CHECKPOINT CHECKSUM CHECK_ACL_REWRITE CHILD CHOOSE CHR CHUNK CLASS CLASSIFICATION
        CLASSIFIER CLAUSE CLEAN CLEANUP CLEAR CLIENT CLONE CLOSE CLOSEST CLOSE_CACHED_OPEN_CURSORS
        CLOUD_IDENTITY CLUSTERING CLUSTERING_FACTOR CLUSTERS CLUSTER_BY_ROWID CLUSTER_DETAILS
        CLUSTER_DISTANCE CLUSTER_ID CLUSTER_PROBABILITY CLUSTER_SET COALESCE COALESCE_SQ
        COARSE COLAUTH COLD COLLATE COLLATION COLLECT COLUMN COLUMNAR COLUMNS COLUMN_AUTHORIZATION_INDICATOR
        COLUMN_AUTH_INDICATOR COLUMN_STATS COLUMN_VALUE COMMENT COMMIT COMMITTED COMMON
        COMMON_DATA_MAP COMPACT COMPATIBILITY COMPILE COMPLETE COMPLIANCE COMPONENT
        COMPONENTS COMPOSE COMPOSITE COMPOSITE_LIMIT COMPOUND COMPUTATION COMPUTE
        CONCAT CONDITION CONDITIONAL CONFIRM CONFORMING CONNECT_BY_CB_WHR_ONLY CONNECT_BY_COMBINE_SW
        CONNECT_BY_COST_BASED CONNECT_BY_ELIM_DUPS CONNECT_BY_FILTERING CONNECT_BY_ISCYCLE
        CONNECT_BY_ISLEAF CONNECT_BY_ROOT CONNECT_TIME CONSENSUS CONSIDER CONSISTENT
        CONST CONSTANT CONSTRAINT CONSTRAINTS CONTAINER CONTAINERS CONTAINERS_DEFAULT
        CONTAINER_DATA CONTAINER_DATA_ADMIT_NULL CONTAINER_MAP CONTAINER_MAP_OBJECT
        CONTENT CONTENTS CONTEXT CONTINUE CONTROLFILE CONVERSION CONVERT CON_DBID_TO_ID
        CON_GUID_TO_ID CON_ID CON_ID_FILTER CON_ID_TO_CON_NAME CON_ID_TO_DBID CON_ID_TO_GUID
        CON_ID_TO_UID CON_NAME_TO_ID CON_UID_TO_ID COOKIE COPY CORR CORRUPTION CORRUPT_XID
        CORRUPT_XID_ALL CORR_K CORR_S COS COSH COST COST_XML_QUERY_REWRITE COUNT COVAR_POP
        COVAR_SAMP CO_AUTH_IND CPU_COSTING CPU_COUNT CPU_PER_CALL CPU_PER_SESSION
        CPU_TIME CRASH CREATE_FILE_DEST CREATE_STORED_OUTLINES CREATION CREDENTIAL
        CREDENTIALS CRITICAL CROSS CROSSEDITION CSCONVERT CUBE CUBE_AJ CUBE_GB CUBE_SJ
        CUME_DIST CUME_DISTM CURRENT CURRENTV CURRENT_DATE CURRENT_INSTANCE CURRENT_PARTSET_KEY
        CURRENT_SCHEMA CURRENT_SHARD_KEY CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER
        CURSOR CURSOR_SHARING_EXACT CURSOR_SPECIFIC_SEGMENT CV CYCLE DAGG_OPTIM_GSETS
        DANGLING DATA DATABASE DATABASES DATAFILE DATAFILES DATAMOVEMENT DATAOBJNO
        DATAOBJ_TO_MAT_PARTITION DATAOBJ_TO_PARTITION DATAPUMP DATASTORE DATA_LINK_DML
        DATA_SECURITY_REWRITE_LIMIT DATA_VALIDATE DATE_MODE DAYS DBA DBA_RECYCLEBIN
        DBMS_STATS DBSTR2UTF8 DBTIMEZONE DB_ROLE_CHANGE DB_UNIQUE_NAME DB_VERSION
        DDL DEALLOCATE DEBUG DEBUGGER DECLARE DECODE DECOMPOSE DECOMPRESS DECORRELATE
        DECR DECREMENT DECRYPT DEDUPLICATE DEFAULTS DEFAULT_COLLATION DEFAULT_PDB_HINT
        DEFERRABLE DEFERRED DEFINE DEFINED DEFINER DEFINITION DEGREE DELAY DELEGATE
        DELETEXML DELETE_ALL DEMAND DENORM_AV DENSE_RANK DENSE_RANKM DEPENDENT DEPTH
        DEQUEUE DEREF DEREF_NO_REWRITE DESCENDANT DESCRIPTION DESTROY DETACHED DETERMINED
        DETERMINES DETERMINISTIC DG_GATHER_STATS DIAGNOSTICS DICTIONARY DIGEST DIMENSION
        DIMENSIONS DIRECT DIRECTORY DIRECT_LOAD DIRECT_PATH DISABLE DISABLE_ALL DISABLE_CONTAINERS_DEFAULT
        DISABLE_CONTAINER_MAP DISABLE_PARALLEL_DML DISABLE_PRESET DISABLE_RPKE DISALLOW
        DISASSOCIATE DISCARD DISCONNECT DISK DISKGROUP DISKS DISMOUNT DISTINGUISHED
        DISTRIBUTE DISTRIBUTED DIST_AGG_PROLLUP_PUSHDOWN DML DML_UPDATE DOCFIDELITY
        DOCUMENT DOMAIN_INDEX_FILTER DOMAIN_INDEX_NO_SORT DOMAIN_INDEX_SORT DOWNGRADE
        DRAIN_TIMEOUT DRIVING_SITE DROP_COLUMN DROP_GROUP DST_UPGRADE_INSERT_CONV
        DUMP DUPLICATE DUPLICATED DV DYNAMIC DYNAMIC_SAMPLING DYNAMIC_SAMPLING_EST_CDN
        EACH EDITION EDITIONABLE EDITIONING EDITIONS ELAPSED_TIME ELEMENT ELIMINATE_JOIN
        ELIMINATE_OBY ELIMINATE_OUTER_JOIN ELIMINATE_SQ ELIM_GROUPBY EM EMPTY EMPTY_BLOB
        EMPTY_CLOB ENABLE ENABLE_ALL ENABLE_PARALLEL_DML ENABLE_PRESET ENCODE ENCODING
        ENCRYPT ENCRYPTION END END_OUTLINE_DATA ENFORCE ENFORCED ENQUEUE ENTERPRISE
        ENTITYESCAPING ENTRY EQUIPART EQUIVALENT ERROR ERRORS ERROR_ARGUMENT ERROR_ON_OVERLAP_TIME
        ESCAPE ESTIMATE EVAL EVALNAME EVALUATE EVALUATION EVEN EVENTS EVERY EXCEPTION
        EXCEPTIONS EXCHANGE EXCLUDE EXCLUDING EXECUTE EXEMPT EXISTING EXISTSNODE EXP
        EXPAND EXPAND_GSET_TO_UNION EXPAND_TABLE EXPIRE EXPLAIN EXPLOSION EXPORT EXPRESS
        EXPR_CORR_CHECK EXTEND EXTENDED EXTENDS EXTENT EXTENTS EXTERNAL EXTERNALLY
        EXTRA EXTRACT EXTRACTCLOBXML EXTRACTVALUE FACILITY FACT FACTOR FACTORIZE_JOIN
        FAILED FAILED_LOGIN_ATTEMPTS FAILGROUP FAILOVER FAILURE FALSE FAMILY FAR FAST
        FBTSCAN FEATURE FEATURE_COMPARE FEATURE_DETAILS FEATURE_ID FEATURE_SET FEATURE_VALUE
        FEDERATION FETCH FILE FILEGROUP FILESTORE FILESYSTEM_LIKE_LOGGING FILE_NAME_CONVERT
        FILTER FINAL FINE FINISH FIRST FIRSTM FIRST_ROWS FIRST_VALUE FIXED_VIEW_DATA
        FLAGGER FLASHBACK FLASH_CACHE FLEX FLOB FLOOR FLUSH FOLDER FOLLOWING FOLLOWS
        FORCE FORCE_JSON_TABLE_TRANSFORM FORCE_SPATIAL FORCE_XML_QUERY_REWRITE FOREIGN
        FOREVER FORMAT FORWARD FRAGMENT_NUMBER FREE FREELIST FREELISTS FREEPOOLS FRESH
        FRESH_MV FROM_TZ FTP FULL FULL_OUTER_JOIN_TO_OUTER FUNCTION FUNCTIONS GATHER_OPTIMIZER_STATISTICS
        GATHER_PLAN_STATISTICS GBY_CONC_ROLLUP GBY_PUSHDOWN GENERATED GET GLOBAL GLOBALLY
        GLOBAL_NAME GLOBAL_TOPIC_ENABLED GOLDENGATE GOTO GRANTED GRANULAR GREATEST
        GROUPING GROUPING_ID GROUPS GROUP_BY GROUP_ID GUARANTEE GUARANTEED GUARD H
        HALF_YEARS HASH HASHING HASHKEYS HASHSET_BUILD HASH_AJ HASH_SJ HEADER HEAP
        HELP HEXTORAW HEXTOREF HIDDEN HIDE HIERARCHICAL HIERARCHIES HIERARCHY HIER_ANCESTOR
        HIER_CAPTION HIER_CHILDREN HIER_CHILD_COUNT HIER_COLUMN HIER_CONDITION HIER_DEPTH
        HIER_DESCRIPTION HIER_HAS_CHILDREN HIER_LAG HIER_LEAD HIER_LEVEL HIER_MEMBER_NAME
        HIER_MEMBER_UNIQUE_NAME HIER_ORDER HIER_PARENT HIER_WINDOW HIGH HINTSET_BEGIN
        HINTSET_END HOST HOT HOUR HOURS HTTP HWM_BROKERED HYBRID ID IDENTIFIER IDENTITY
        IDGENERATORS IDLE IDLE_TIME IF IGNORE IGNORE_OPTIM_EMBEDDED_HINTS IGNORE_ROW_ON_DUPKEY_INDEX
        IGNORE_WHERE_CLAUSE ILM IMMEDIATE IMMUTABLE IMPACT IMPORT INACTIVE INACTIVE_ACCOUNT_TIME
        INCLUDE INCLUDES INCLUDE_VERSION INCLUDING INCOMING INCR INCREMENT INCREMENTAL
        INDENT INDEXED INDEXES INDEXING INDEXTYPE INDEXTYPES INDEX_ASC INDEX_COMBINE
        INDEX_DESC INDEX_FFS INDEX_FILTER INDEX_JOIN INDEX_ROWS INDEX_RRS INDEX_RS
        INDEX_RS_ASC INDEX_RS_DESC INDEX_SCAN INDEX_SKIP_SCAN INDEX_SS INDEX_SS_ASC
        INDEX_SS_DESC INDEX_STATS INDICATOR INFINITE INFORMATIONAL INHERIT INITCAP
        INITIAL INITIALIZED INITIALLY INITRANS INLINE INLINE_XMLTYPE_NT INLINE_XT
        INMEMORY INMEMORY_PRUNING INNER INPLACE INSENSITIVE INSERTCHILDXML INSERTCHILDXMLAFTER
        INSERTCHILDXMLBEFORE INSERTXMLAFTER INSERTXMLBEFORE INSTALL INSTANCE INSTANCES
        INSTANTIABLE INSTANTLY INSTEAD INSTR INSTR2 INSTR4 INSTRB INSTRC INTERLEAVED
        INTERMEDIATE INTERNAL_CONVERT INTERNAL_USE INTERPRETED INTRA_CDB INVALIDATE
        INVALIDATION INVISIBLE IN_MEMORY_METADATA IN_XQUERY IOSEEKTIM IOTFRSPEED IO_LOGICAL
        IO_MEGABYTES IO_REQUESTS ISOLATE ISOLATION ISOLATION_LEVEL ITERATE ITERATION_NUMBER
        JAVA JOB JOIN JSON JSONGET JSONPARSE JSONTOXML JSON_ARRAY JSON_ARRAYAGG JSON_EQUAL
        JSON_EQUAL2 JSON_EXISTS JSON_EXISTS2 JSON_HASH JSON_LENGTH JSON_MERGEPATCH
        JSON_MKMVI JSON_OBJECT JSON_OBJECTAGG JSON_PATCH JSON_QUERY JSON_SCALAR JSON_SERIALIZE
        JSON_TABLE JSON_TEXTCONTAINS JSON_TEXTCONTAINS2 JSON_TRANSFORM JSON_VALUE
        KEEP KEEP_DUPLICATES KERBEROS KEY KEYS KEYSIZE KEYSTORE KEY_LENGTH KILL
        KURTOSIS_POP KURTOSIS_SAMP LABEL LAG LAG_DIFF LAG_DIFF_PERCENT LANGUAGE LAST
        LAST_DAY LAST_VALUE LATERAL LAX LAYER LDAP_REGISTRATION LDAP_REGISTRATION_ENABLED
        LDAP_REG_SYNC_INTERVAL LEAD LEADING LEAD_CDB LEAD_CDB_URI LEAD_DIFF LEAD_DIFF_PERCENT
        LEAF LEAST LEAVES LEDGER LEFT LENGTH LENGTH2 LENGTH4 LENGTHB LENGTHC LESS
        LEVEL LEVELS LIBRARY LIFE LIFECYCLE LIFETIME LIKE2 LIKE4 LIKEC LIMIT LINEAR
        LINK LIST LISTAGG LN LNNVL LOAD LOB LOBNVL LOBS LOB_VALUE LOCALTIME LOCALTIMESTAMP
        LOCAL_INDEXES LOCATION LOCATOR LOCKDOWN LOCKED LOCKING LOG LOGFILE LOGFILES
        LOGGING LOGICAL LOGICAL_READS_PER_CALL LOGICAL_READS_PER_SESSION LOGMINING
        LOGOFF LOGON LOG_READ_ONLY_VIOLATIONS LOST LOW LOWER LPAD LTRIM MAIN MAKE_REF
        MANAGE MANAGED MANAGEMENT MANAGER MANDATORY MANUAL MAP MAPPER MAPPING MASTER
        MATCH MATCHED MATCHES MATCH_NUMBER MATCH_RECOGNIZE MATERIALIZE MATERIALIZED
        MATRIX MAX MAXARCHLOGS MAXDATAFILES MAXEXTENTS MAXIMIZE MAXINSTANCES MAXLOGFILES
        MAXLOGHISTORY MAXLOGMEMBERS MAXSIZE MAXTRANS MAXVALUE MAX_AUDIT_SIZE MAX_DIAG_SIZE
        MAX_PDB_SNAPSHOTS MAX_SHARED_TEMP_SIZE MBRC MEASURE MEASURES MEDIAN MEDIUM
        MEMBER MEMCOMPRESS MEMOPTIMIZE MEMOPTIMIZE_WRITE MEMORY MERGE MERGE$ACTIONS
        MERGE_AJ MERGE_CONST_ON MERGE_SJ METADATA METADATA_SOURCE_PDB METHOD MIGRATE
        MIGRATE_CROSS_CON MIGRATION MIN MINEXTENTS MINIMIZE MINIMUM MINING MINUS_NULL
        MINUTE MINUTES MINVALUE MIRROR MIRRORCOLD MIRRORHOT MISMATCH MISSING MLE MLSLABEL
        MOD MODEL MODEL_COMPILE_SUBQUERY MODEL_DONTVERIFY_UNIQUENESS MODEL_DYNAMIC_SUBQUERY
        MODEL_MIN_ANALYSIS MODEL_NB MODEL_NO_ANALYSIS MODEL_PBY MODEL_PUSH_REF MODEL_SV
        MODIFICATION MODIFY MODIFY_COLUMN_TYPE MODULE MONITOR MONITORING MONTHS MONTHS_BETWEEN
        MOUNT MOUNTPATH MOUNTPOINT MOVE MOVEMENT MULTIDIMENSIONAL MULTISET MULTIVALUE
        MV_MERGE NAME NAMED NAMES NAMESPACE NAN NANVL NATIVE NATIVE_FULL_OUTER_JOIN
        NATURAL NAV NCHAR_CS NCHR NEEDED NEG NESTED NESTED_ROLLUP_TOP NESTED_TABLE_FAST_INSERT
        NESTED_TABLE_GET_REFS NESTED_TABLE_ID NESTED_TABLE_SET_REFS NESTED_TABLE_SET_SETID
        NETWORK NEVER NEW NEW_TIME NEXT NEXT_DAY NLJ_BATCHING NLJ_INDEX_FILTER NLJ_INDEX_SCAN
        NLJ_PREFETCH NLSSORT NLS_CALENDAR NLS_CHARACTERSET NLS_CHARSET_DECL_LEN NLS_CHARSET_ID
        NLS_CHARSET_NAME NLS_COLLATION_ID NLS_COLLATION_NAME NLS_COMP NLS_CURRENCY
        NLS_DATE_FORMAT NLS_DATE_LANGUAGE NLS_INITCAP NLS_ISO_CURRENCY NLS_LANG NLS_LANGUAGE
        NLS_LENGTH_SEMANTICS NLS_LOWER NLS_NCHAR_CONV_EXCP NLS_NUMERIC_CHARACTERS
        NLS_SORT NLS_SPECIAL_CHARS NLS_TERRITORY NLS_UPPER NL_AJ NL_SJ NO NOAPPEND
        NOARCHIVELOG NOAUDIT NOCACHE NOCOPY NOCPU_COSTING NOCYCLE NODELAY NOENTITYESCAPING
        NOEXTEND NOFORCE NOGUARANTEE NOKEEP NOLOCAL NOLOGGING NOMAPPING NOMAXVALUE
        NOMINIMIZE NOMINVALUE NOMONITORING NONBLOCKING NONE NONEDITIONABLE NONPARTITIONED
        NONSCHEMA NOORDER NOOVERRIDE NOPARALLEL NOPARALLEL_INDEX NORELOCATE NORELY
        NOREPAIR NOREPLAY NORESETLOGS NOREVERSE NOREWRITE NORMAL NOROWDEPENDENCIES
        NOSCALE NOSCHEMACHECK NOSEGMENT NOSHARD NOSORT NOSTRICT NOSWITCH NOTHING NOTIFICATION
        NOVALIDATE NOW NO_ACCESS NO_ADAPTIVE_PLAN NO_ANSI_REARCH NO_ANSWER_QUERY_USING_STATS
        NO_AUTO_REOPTIMIZE NO_BAND_JOIN NO_BASETABLE_MULTIMV_REWRITE NO_BATCH_TABLE_ACCESS_BY_ROWID
        NO_BIND_AWARE NO_BUFFER NO_BUSHY_JOIN NO_CARTESIAN NO_CHECK_ACL_REWRITE NO_CLUSTERING
        NO_CLUSTER_BY_ROWID NO_COALESCE_SQ NO_COMMON_DATA NO_CONNECT_BY_CB_WHR_ONLY
        NO_CONNECT_BY_COMBINE_SW NO_CONNECT_BY_COST_BASED NO_CONNECT_BY_ELIM_DUPS
        NO_CONNECT_BY_FILTERING NO_CONTAINERS NO_COST_XML_QUERY_REWRITE NO_CPU_COSTING
        NO_CROSS_CONTAINER NO_DAGG_OPTIM_GSETS NO_DATA_SECURITY_REWRITE NO_DECORRELATE
        NO_DIST_AGG_PROLLUP_PUSHDOWN NO_DOMAIN_INDEX_FILTER NO_DST_UPGRADE_INSERT_CONV
        NO_ELIMINATE_JOIN NO_ELIMINATE_OBY NO_ELIMINATE_OUTER_JOIN NO_ELIMINATE_SQ
        NO_ELIM_GROUPBY NO_EXPAND NO_EXPAND_GSET_TO_UNION NO_EXPAND_TABLE NO_FACT
        NO_FACTORIZE_JOIN NO_FILTERING NO_FULL_OUTER_JOIN_TO_OUTER NO_GATHER_OPTIMIZER_STATISTICS
        NO_GBY_PUSHDOWN NO_INDEX NO_INDEX_FFS NO_INDEX_SS NO_INMEMORY NO_INMEMORY_PRUNING
        NO_JSON_TABLE_TRANSFORM NO_LOAD NO_MERGE NO_MODEL_PUSH_REF NO_MONITOR NO_MONITORING
        NO_MULTIMV_REWRITE NO_NATIVE_FULL_OUTER_JOIN NO_NLJ_BATCHING NO_NLJ_PREFETCH
        NO_OBJECT_LINK NO_OBY_GBYPD_SEPARATE NO_ORDER_ROLLUPS NO_OR_EXPAND NO_OUTER_JOIN_TO_ANTI
        NO_OUTER_JOIN_TO_INNER NO_PARALLEL NO_PARALLEL_INDEX NO_PARTIAL_COMMIT NO_PARTIAL_JOIN
        NO_PARTIAL_OSON_UPDATE NO_PARTIAL_ROLLUP_PUSHDOWN NO_PLACE_DISTINCT NO_PLACE_GROUP_BY
        NO_PQ_CONCURRENT_UNION NO_PQ_EXPAND_TABLE NO_PQ_MAP NO_PQ_NONLEAF_SKEW NO_PQ_REPLICATE
        NO_PQ_SKEW NO_PRUNE_GSETS NO_PULL_PRED NO_PUSH_HAVING_TO_GBY NO_PUSH_PRED
        NO_PUSH_SUBQ NO_PX_FAULT_TOLERANCE NO_PX_JOIN_FILTER NO_QKN_BUFF NO_QUERY_TRANSFORMATION
        NO_REF_CASCADE NO_REORDER_WIF NO_RESULT_CACHE NO_REWRITE NO_ROOT_SW_FOR_LOCAL
        NO_SEMIJOIN NO_SEMI_TO_INNER NO_SET_GBY_PUSHDOWN NO_SET_TO_JOIN NO_SQL_TRANSLATION
        NO_SQL_TUNE NO_STAR_TRANSFORMATION NO_STATEMENT_QUEUING NO_STATS_GSETS NO_SUBQUERY_PRUNING
        NO_SUBSTRB_PAD NO_SWAP_JOIN_INPUTS NO_TABLE_LOOKUP_BY_NL NO_TEMP_TABLE NO_TRANSFORM_DISTINCT_AGG
        NO_UNNEST NO_USE_CUBE NO_USE_DAGG_UNION_ALL_GSETS NO_USE_HASH NO_USE_HASH_AGGREGATION
        NO_USE_HASH_GBY_FOR_DAGGPSHD NO_USE_HASH_GBY_FOR_PUSHDOWN NO_USE_INVISIBLE_INDEXES
        NO_USE_MERGE NO_USE_NL NO_USE_PARTITION_WISE_DISTINCT NO_USE_PARTITION_WISE_GBY
        NO_USE_PARTITION_WISE_WIF NO_USE_SCALABLE_GBY_INVDIST NO_USE_VECTOR_AGGREGATION
        NO_VECTOR_TRANSFORM NO_VECTOR_TRANSFORM_DIMS NO_VECTOR_TRANSFORM_FACT NO_XDB_FASTPATH_INSERT
        NO_XMLINDEX_REWRITE NO_XMLINDEX_REWRITE_IN_SELECT NO_XML_DML_REWRITE NO_XML_QUERY_REWRITE
        NO_ZONEMAP NTH_VALUE NTILE NULLIF NULLS NUMTODSINTERVAL NUMTOYMINTERVAL NUM_INDEX_KEYS
        NVL NVL2 OBJECT OBJECT2XML OBJNO OBJNO_REUSE OBJ_ID OBY_GBYPD_SEPARATE OCCURENCES
        OCCURRENCES ODD OFF OFFLINE OFFSET OID OIDINDEX OLAP OLD OLD_PUSH_PRED OLS
        OLTP OMIT ONE ONLINE ONLY OPAQUE OPAQUE_TRANSFORM OPAQUE_XCANONICAL OPCODE
        OPEN OPERATIONS OPERATOR OPTIMAL OPTIMIZE OPTIMIZER_FEATURES_ENABLE OPTIMIZER_GOAL
        OPT_ESTIMATE OPT_PARAM ORADEBUG ORA_BRANCH ORA_CHECK_ACL ORA_CHECK_PRIVILEGE
        ORA_CHECK_SYS_PRIVILEGE ORA_CLUSTERING ORA_CONCAT_RWKEY ORA_DM_PARTITION_NAME
        ORA_DST_AFFECTED ORA_DST_CONVERT ORA_DST_ERROR ORA_GET_ACLIDS ORA_GET_PRIVILEGES
        ORA_HASH ORA_INVOKING_USER ORA_INVOKING_USERID ORA_INVOKING_XS_USER ORA_INVOKING_XS_USER_GUID
        ORA_NORMALIZE ORA_PARTITION_VALIDATION ORA_RAWCOMPARE ORA_RAWCONCAT ORA_ROWSCN
        ORA_ROWSCN_RAW ORA_ROWVERSION ORA_SEARCH_RWKEY ORA_SHARDSPACE_NAME ORA_SHARD_ID
        ORA_TABVERSION ORA_WRITE_TIME ORDERED ORDERED_PREDICATES ORDER_KEY_VECTOR_USE
        ORDER_SUBQ ORDINALITY ORGANIZATION OR_EXPAND OR_PREDICATES OSON OSON_DIAG
        OSON_GET_CONTENT OTHER OTHERS OUTER OUTER_JOIN_TO_ANTI OUTER_JOIN_TO_INNER
        OUTLINE OUTLINE_LEAF OUT_OF_LINE OVER OVERFLOW OVERFLOW_NOMOVE OVERLAPS OWN
        OWNER OWNERSHIP PACKAGE PACKAGES PARALLEL PARALLEL_INDEX PARAM PARAMETERS
        PARENT PARITY PART$NUM$INST PARTIAL PARTIALLY PARTIAL_JOIN PARTIAL_ROLLUP_PUSHDOWN
        PARTITION PARTITIONING PARTITIONS PARTITIONSET PARTITION_CONTAINED PARTITION_HASH
        PARTITION_LIST PARTITION_RANGE PASSING PASSIVE PASSWORD PASSWORDFILE_METADATA_CACHE
        PASSWORD_GRACE_TIME PASSWORD_LIFE_TIME PASSWORD_LOCK_TIME PASSWORD_REUSE_MAX
        PASSWORD_REUSE_TIME PASSWORD_ROLLOVER_TIME PASSWORD_VERIFY_FUNCTION PAST PATCH
        PATH PATHS PATH_PREFIX PATTERN PBL_HS_BEGIN PBL_HS_END PCTINCREASE PCTTHRESHOLD
        PCTUSED PCTVERSION PDB_LOCAL_ONLY PEER PEERS PENDING PER PERCENT PERCENTAGE
        PERCENTILE_CONT PERCENTILE_DISC PERCENT_RANK PERCENT_RANKM PERFORMANCE PERIOD
        PERMANENT PERMISSION PERMUTE PERSISTABLE PFILE PHV PHYSICAL PIKEY PIVOT PIV_GB
        PIV_SSF PLACE_DISTINCT PLACE_GROUP_BY PLAN PLSCOPE_SETTINGS PLSQL_CCFLAGS
        PLSQL_CODE_TYPE PLSQL_DEBUG PLSQL_OPTIMIZE_LEVEL PLSQL_WARNINGS PLUGGABLE
        PMEM POINT POLICY POOL_16K POOL_2K POOL_32K POOL_4K POOL_8K PORT POSITION
        POST_TRANSACTION POWER POWERMULTISET POWERMULTISET_BY_CARDINALITY PQ_CONCURRENT_UNION
        PQ_DISTRIBUTE PQ_DISTRIBUTE_WINDOW PQ_EXPAND_TABLE PQ_FILTER PQ_MAP PQ_NOMAP
        PQ_NONLEAF_SKEW PQ_REPLICATE PQ_SKEW PREBUILT PRECEDES PRECEDING PRECOMPUTE_SUBQUERY
        PREDICATE_REORDERS PREDICTION PREDICTION_BOUNDS PREDICTION_COST PREDICTION_DETAILS
        PREDICTION_PROBABILITY PREDICTION_SET PRELOAD PREPARE PRESENT PRESENTNNV PRESENTV
        PRESERVE PRESERVE_OID PRETTY PREV PREVIOUS PRIMARY PRINTBLOBTOCLOB PRIORITY
        PRIVATE PRIVATE_SGA PRIVILEGE PRIVILEGED PRIVILEGES PROCEDURAL PROCEDURE PROCESS
        PROFILE PROGRAM PROJECT PROPAGATE PROPAGATION PROPERTY PROTECTED PROTECTION
        PROTOCOL PROXY PRUNING PULL_PRED PURGE PUSH_HAVING_TO_GBY PUSH_PRED PUSH_SUBQ
        PX_FAULT_TOLERANCE PX_GRANULE PX_JOIN_FILTER QB_NAME QUALIFY QUARANTINE QUARTERS
        QUERY QUERY_BLOCK QUEUE QUEUE_CURR QUEUE_ROWP QUIESCE QUORUM QUOTA QUOTAGROUP
        QUOTES RANDOM RANDOM_LOCAL RANGE RANK RANKM RAPIDLY RATIO_TO_REPORT RAWTOHEX
        RAWTONHEX RAWTOREF RBA RBO_OUTLINE RDBA READ READS READ_OR_WRITE REALM REBALANCE
        REBUILD RECONNECT RECORDS_PER_BLOCK RECOVER RECOVERABLE RECOVERY RECYCLE RECYCLEBIN
        REDACTION REDEFINE REDO REDUCED REDUNDANCY REFERENCE REFERENCED REFERENCES
        REFERENCING REFRESH REFTOHEX REFTORAW REF_CASCADE_CURSOR REGEXP_COUNT REGEXP_INSTR
        REGEXP_LIKE REGEXP_REPLACE REGEXP_SUBSTR REGISTER REGR_AVGX REGR_AVGY REGR_COUNT
        REGR_INTERCEPT REGR_R2 REGR_SLOPE REGR_SXX REGR_SXY REGR_SYY REGULAR REJECT
        REKEY RELATIONAL RELOCATE RELY REMAINDER REMOTE REMOTE_MAPPED REMOVE REORDER_WIF
        REPAIR REPEAT REPLACE REPLICATION REQUIRED RESERVOIR_SAMPLING RESET RESETLOGS
        RESIZE RESOLVE RESOLVER RESPECT RESTART RESTORE RESTORE_AS_INTERVALS RESTRICT
        RESTRICTED RESTRICT_ALL_REF_CONS RESULT_CACHE RESUMABLE RESUME RETENTION RETRY_ON_ROW_CHANGE
        RETURN RETURNING REUSE REVERSE REWRITE REWRITE_OR_ERROR RIGHT RLS_FORCE ROLE
        ROLES ROLESET ROLLBACK ROLLING ROLLOVER ROLLUP ROOT ROUND ROUND_TIES_TO_EVEN
        ROW ROWDEPENDENCIES ROWIDTOCHAR ROWIDTONCHAR ROWID_MAPPING_TABLE ROWNUM ROWS
        ROW_LENGTH ROW_NUMBER RPAD RTRIM RULE RULES RUNNING SALT SAMPLE SAVE SAVEPOINT
        SAVE_AS_INTERVALS SB4 SCALAR SCALARS SCALE SCALE_ROWS SCAN SCAN_INSTANCES
        SCHEDULER SCHEMA SCHEMACHECK SCN SCN_ASCENDING SCOPE SCRUB SDO_GEOM_KEY SDO_GEOM_MAX_X
        SDO_GEOM_MAX_Y SDO_GEOM_MAX_Z SDO_GEOM_MBB SDO_GEOM_MBR SDO_GEOM_MIN_X SDO_GEOM_MIN_Y
        SDO_GEOM_MIN_Z SDO_TOLERANCE SD_ALL SD_INHIBIT SD_SHOW SEARCH SECONDS SECRET
        SECUREFILE SECUREFILE_DBA SECURITY SEED SEGMENT SEG_BLOCK SEG_FILE SELECTIVITY
        SELF SEMIJOIN SEMIJOIN_DRIVER SEMI_TO_INNER SENSITIVE SEQUENCE SEQUENCED SEQUENTIAL
        SERIAL SERIALIZABLE SERVERERROR SERVICE SERVICES SERVICE_NAME_CONVERT SESSION
        SESSIONS_PER_USER SESSIONTIMEZONE SESSIONTZNAME SESSION_CACHED_CURSORS SETS
        SETTINGS SET_GBY_PUSHDOWN SET_TO_JOIN SEVERE SHARD SHARDED SHARDS SHARDSPACE
        SHARD_CHUNK_ID SHARED SHARED_POOL SHARE_OF SHARING SHD$COL$MAP SHELFLIFE SHOW
        SHRINK SHUTDOWN SIBLING SIBLINGS SID SIGN SIGNAL_COMPONENT SIGNAL_FUNCTION
        SIGNATURE SIMPLE SIN SINGLE SINGLETASK SINH SITE SKEWNESS_POP SKEWNESS_SAMP
        SKIP SKIP_EXT_OPTIMIZER SKIP_PROXY SKIP_UNQ_UNUSABLE_IDX SKIP_UNUSABLE_INDEXES
        SMALLFILE SNAPSHOT SOME SORT SOUNDEX SOURCE SOURCE_FILE_DIRECTORY SOURCE_FILE_NAME_CONVERT
        SPACE SPATIAL SPECIFICATION SPFILE SPLIT SPREADSHEET SQL SQLLDR SQL_SCOPE
        SQL_TRACE SQL_TRANSLATION_PROFILE SQRT STALE STANDALONE STANDARD_HASH STANDBY
        STANDBYS STANDBY_MAX_DATA_DELAY STAR STARTUP STAR_TRANSFORMATION STATE STATEMENT
        STATEMENTS STATEMENT_ID STATEMENT_QUEUING STATIC STATISTICS STATS_BINOMIAL_TEST
        STATS_CROSSTAB STATS_F_TEST STATS_KS_TEST STATS_MODE STATS_MW_TEST STATS_ONE_WAY_ANOVA
        STATS_T_TEST_INDEP STATS_T_TEST_INDEPU STATS_T_TEST_ONE STATS_T_TEST_PAIRED
        STATS_WSR_TEST STDDEV STDDEV_POP STDDEV_SAMP STOP STORAGE STORAGE_INDEX STORE
        STREAM STREAMS STRICT STRING STRINGS STRIP STRIPE_COLUMNS STRIPE_WIDTH STRUCTURE
        SUBMULTISET SUBPARTITION SUBPARTITIONING SUBPARTITIONS SUBPARTITION_REL SUBQUERIES
        SUBQUERY_PRUNING SUBSCRIBE SUBSET SUBSTITUTABLE SUBSTR SUBSTR2 SUBSTR4 SUBSTRB
        SUBSTRC SUBTYPE SUCCESS SUCCESSFUL SUM SUMMARY SUPPLEMENTAL SUPPRESS_LOAD
        SUSPEND SWAP_JOIN_INPUTS SWITCH SWITCHOVER SYNC SYNCHRONOUS SYSASM SYSAUX
        SYSBACKUP SYSDATE SYSDBA SYSDG SYSGUID SYSKM SYSOBJ SYSOPER SYSRAC SYSTEM
        SYSTEM_DEFINED SYSTEM_STATS SYSTIMESTAMP SYS_AUDIT SYS_CHECKACL SYS_CHECK_PRIVILEGE
        SYS_CONNECT_BY_PATH SYS_CONS_ANY_SCALAR SYS_CONTEXT SYS_CTXINFOPK SYS_CTX_CONTAINS2
        SYS_CTX_MKIVIDX SYS_DBURIGEN SYS_DL_CURSOR SYS_DM_RXFORM_CHR SYS_DM_RXFORM_LAB
        SYS_DM_RXFORM_NUM SYS_DOM_COMPARE SYS_DST_PRIM2SEC SYS_DST_SEC2PRIM SYS_ET_BFILE_TO_RAW
        SYS_ET_BLOB_TO_IMAGE SYS_ET_IMAGE_TO_BLOB SYS_ET_RAW_TO_BFILE SYS_EXTPDTXT
        SYS_EXTRACT_UTC SYS_FBT_INSDEL SYS_FILTER_ACLS SYS_FNMATCHES SYS_FNREPLACE
        SYS_GETTOKENID SYS_GETXTIVAL SYS_GET_ACLIDS SYS_GET_ANY_SCALAR SYS_GET_COL_ACLIDS
        SYS_GET_PRIVILEGES SYS_GUID SYS_MAKEXML SYS_MAKE_XMLNODEID SYS_MKXMLATTR SYS_MKXTI
        SYS_OPTLOBPRBSC SYS_OPTXICMP SYS_OPTXQCASTASNQ SYS_OP_ADT2BIN SYS_OP_ADTCONS
        SYS_OP_ALSCRVAL SYS_OP_ATG SYS_OP_BIN2ADT SYS_OP_BITVEC SYS_OP_BL2R SYS_OP_BLOOM_FILTER
        SYS_OP_BLOOM_FILTER_LIST SYS_OP_C2C SYS_OP_CAST SYS_OP_CEG SYS_OP_CL2C SYS_OP_COMBINED_HASH
        SYS_OP_COMP SYS_OP_CONVERT SYS_OP_COUNTCHG SYS_OP_CSCONV SYS_OP_CSCONVTEST
        SYS_OP_CSR SYS_OP_CSX_PATCH SYS_OP_CYCLED_SEQ SYS_OP_DECOMP SYS_OP_DESCEND
        SYS_OP_DISTINCT SYS_OP_DRA SYS_OP_DSB_DESERIALIZE SYS_OP_DSB_SERIALIZE SYS_OP_DUMP
        SYS_OP_DV_CHECK SYS_OP_ENFORCE_NOT_NULL$ SYS_OP_EXTRACT SYS_OP_GROUPING SYS_OP_GUID
        SYS_OP_HASH SYS_OP_HCS_TABLE SYS_OP_IIX SYS_OP_INTERVAL_HIGH_BOUND SYS_OP_ITR
        SYS_OP_KEY_VECTOR_CREATE SYS_OP_KEY_VECTOR_FILTER SYS_OP_KEY_VECTOR_FILTER_LIST
        SYS_OP_KEY_VECTOR_PAYLOAD SYS_OP_KEY_VECTOR_SUCCEEDED SYS_OP_KEY_VECTOR_USE
        SYS_OP_LBID SYS_OP_LOBLOC2BLOB SYS_OP_LOBLOC2CLOB SYS_OP_LOBLOC2ID SYS_OP_LOBLOC2NCLOB
        SYS_OP_LOBLOC2TYP SYS_OP_LSVI SYS_OP_LVL SYS_OP_MAKEOID SYS_OP_MAP_NONNULL
        SYS_OP_MSR SYS_OP_NICOMBINE SYS_OP_NIEXTRACT SYS_OP_NII SYS_OP_NIX SYS_OP_NOEXPAND
        SYS_OP_NTCIMG$ SYS_OP_NUMTORAW SYS_OP_OBJ_UPD_IN_TXN SYS_OP_OIDVALUE SYS_OP_OPNSIZE
        SYS_OP_PAR SYS_OP_PARGID SYS_OP_PARGID_1 SYS_OP_PART_ID SYS_OP_PAR_1 SYS_OP_PIVOT
        SYS_OP_R2O SYS_OP_RAWTONUM SYS_OP_RDTM SYS_OP_REF SYS_OP_RMTD SYS_OP_ROWIDTOOBJ
        SYS_OP_RPB SYS_OP_TOSETID SYS_OP_TPR SYS_OP_TRTB SYS_OP_UNDESCEND SYS_OP_VECAND
        SYS_OP_VECBIT SYS_OP_VECOR SYS_OP_VECTOR_GROUP_BY SYS_OP_VECXOR SYS_OP_VERSION
        SYS_OP_VREF SYS_OP_VVD SYS_OP_XMLCONS_FOR_CSX SYS_OP_XPTHATG SYS_OP_XPTHIDX
        SYS_OP_XPTHOP SYS_OP_XTNN SYS_OP_XTXT2SQLT SYS_OP_ZONE_ID SYS_ORDERKEY_DEPTH
        SYS_ORDERKEY_MAXCHILD SYS_ORDERKEY_PARENT SYS_PARALLEL_TXN SYS_PATHID_IS_ATTR
        SYS_PATHID_IS_NMSPC SYS_PATHID_LASTNAME SYS_PATHID_LASTNMSPC SYS_PATH_REVERSE
        SYS_PLSQL_COUNT SYS_PLSQL_CPU SYS_PLSQL_IO SYS_PXQEXTRACT SYS_RAW_TO_XSID
        SYS_REMAP_XMLTYPE SYS_RID_ORDER SYS_ROW_DELTA SYS_SC_2_XMLT SYS_SYNRCIREDO
        SYS_TYPEID SYS_UMAKEXML SYS_XMLANALYZE SYS_XMLCONTAINS SYS_XMLCONV SYS_XMLEXNSURI
        SYS_XMLGEN SYS_XMLINSTR SYS_XMLI_LOC_ISNODE SYS_XMLI_LOC_ISTEXT SYS_XMLLOCATOR_GETSVAL
        SYS_XMLNODEID SYS_XMLNODEID_GETCID SYS_XMLNODEID_GETLOCATOR SYS_XMLNODEID_GETOKEY
        SYS_XMLNODEID_GETPATHID SYS_XMLNODEID_GETPTRID SYS_XMLNODEID_GETRID SYS_XMLNODEID_GETSVAL
        SYS_XMLNODEID_GETTID SYS_XMLTRANSLATE SYS_XMLTYPE2SQL SYS_XMLT_2_SC SYS_XQBASEURI
        SYS_XQCASTABLEERRH SYS_XQCODEP2STR SYS_XQCODEPEQ SYS_XQCON2SEQ SYS_XQCONCAT
        SYS_XQDELETE SYS_XQDFLTCOLATION SYS_XQDOC SYS_XQDOCURI SYS_XQDURDIV SYS_XQED4URI
        SYS_XQENDSWITH SYS_XQERR SYS_XQERRH SYS_XQESHTMLURI SYS_XQEXLOBVAL SYS_XQEXSTWRP
        SYS_XQEXTRACT SYS_XQEXTRREF SYS_XQEXVAL SYS_XQFB2STR SYS_XQFNBOOL SYS_XQFNCMP
        SYS_XQFNDATIM SYS_XQFNLNAME SYS_XQFNNM SYS_XQFNNSURI SYS_XQFNPREDTRUTH SYS_XQFNQNM
        SYS_XQFNROOT SYS_XQFORMATNUM SYS_XQFTCONTAIN SYS_XQFUNCR SYS_XQGETCONTENT
        SYS_XQINDXOF SYS_XQINSERT SYS_XQINSPFX SYS_XQIRI2URI SYS_XQLANG SYS_XQLLNMFRMQNM
        SYS_XQMKNODEREF SYS_XQNILLED SYS_XQNODENAME SYS_XQNORMSPACE SYS_XQNORMUCODE
        SYS_XQNSP4PFX SYS_XQNSPFRMQNM SYS_XQPFXFRMQNM SYS_XQPOLYABS SYS_XQPOLYADD
        SYS_XQPOLYCEL SYS_XQPOLYCST SYS_XQPOLYCSTBL SYS_XQPOLYDIV SYS_XQPOLYFLR SYS_XQPOLYMOD
        SYS_XQPOLYMUL SYS_XQPOLYRND SYS_XQPOLYSQRT SYS_XQPOLYSUB SYS_XQPOLYUMUS SYS_XQPOLYUPLS
        SYS_XQPOLYVEQ SYS_XQPOLYVGE SYS_XQPOLYVGT SYS_XQPOLYVLE SYS_XQPOLYVLT SYS_XQPOLYVNE
        SYS_XQREF2VAL SYS_XQRENAME SYS_XQREPLACE SYS_XQRESVURI SYS_XQRNDHALF2EVN SYS_XQRSLVQNM
        SYS_XQRYENVPGET SYS_XQRYVARGET SYS_XQRYWRP SYS_XQSEQ2CON SYS_XQSEQ2CON4XC
        SYS_XQSEQDEEPEQ SYS_XQSEQINSB SYS_XQSEQRM SYS_XQSEQRVS SYS_XQSEQSUB SYS_XQSEQTYPMATCH
        SYS_XQSTARTSWITH SYS_XQSTATBURI SYS_XQSTR2CODEP SYS_XQSTRJOIN SYS_XQSUBSTRAFT
        SYS_XQSUBSTRBEF SYS_XQTOKENIZE SYS_XQTREATAS SYS_XQXFORM SYS_XQ_ASQLCNV SYS_XQ_ATOMCNVCHK
        SYS_XQ_NRNG SYS_XQ_PKSQL2XML SYS_XQ_UPKXML2SQL SYS_XSID_TO_RAW SYS_ZMAP_FILTER
        SYS_ZMAP_REFRESH TABAUTH TABLES TABLESPACE TABLESPACE_NO TABLE_LOOKUP_BY_NL
        TABLE_STATS TABNO TAG TAN TANH TARGET TBL$OR$IDX$PART$NUM TEMP TEMPFILE TEMPLATE
        TEMPORARY TEMP_TABLE TENANT_ID TEST TEXT THAN THE THREAD THROUGH TIER TIES
        TIMEOUT TIMES TIMESTAMP_TO_NUMBER TIMEZONE_ABBR TIMEZONE_HOUR TIMEZONE_MINUTE
        TIMEZONE_OFFSET TIMEZONE_REGION TIME_ZONE TIV_GB TIV_SSF TOKEN TOPLEVEL TO_ACLID
        TO_APPROX_COUNT_DISTINCT TO_APPROX_PERCENTILE TO_BINARY_DOUBLE TO_BINARY_FLOAT
        TO_BLOB TO_CHAR TO_CLOB TO_DATE TO_DSINTERVAL TO_ISO_STRING TO_LOB TO_MULTI_BYTE
        TO_NCHAR TO_NCLOB TO_NUMBER TO_SINGLE_BYTE TO_TIME TO_TIMESTAMP TO_TIMESTAMP_TZ
        TO_TIME_TZ TO_UTC_TIMESTAMP_TZ TO_YMINTERVAL TRACE TRACING TRACKING TRAILING
        TRANSACTION TRANSFORM TRANSFORM_DISTINCT_AGG TRANSITION TRANSITIONAL TRANSLATE
        TRANSLATION TRANSPORTABLE TREAT TRIGGERS TRIM TRUE TRUNC TRUNCATE TRUST TRUSTED
        TUNING TX TYPE TYPENAME TYPES TZ_OFFSET UB2 UBA UCS2 UID UNARCHIVED UNBOUND
        UNBOUNDED UNCONDITIONAL UNDER UNDO UNDROP UNIFORM UNINSTALL UNION_ALL UNISTR
        UNITE UNIXTIME UNLIMITED UNLOAD UNLOCK UNMATCHED UNNEST UNNEST_INNERJ_DISTINCT_VIEW
        UNNEST_NOSEMIJ_NODISTINCTVIEW UNNEST_SEMIJ_VIEW UNPACKED UNPIVOT UNPLUG UNPROTECTED
        UNQUIESCE UNRECOVERABLE UNRESTRICTED UNSUBSCRIBE UNTIL UNUSABLE UNUSED UPDATABLE
        UPDATED UPDATEXML UPD_INDEXES UPD_JOININDEX UPGRADE UPPER UPSERT USABLE USAGE
        USE USER USERENV USERGROUP USERS USER_DATA USER_DEFINED USER_RECYCLEBIN USER_TABLESPACES
        USE_ANTI USE_CONCAT USE_CUBE USE_DAGG_UNION_ALL_GSETS USE_HASH USE_HASH_AGGREGATION
        USE_HASH_GBY_FOR_DAGGPSHD USE_HASH_GBY_FOR_PUSHDOWN USE_HIDDEN_PARTITIONS
        USE_INVISIBLE_INDEXES USE_MERGE USE_MERGE_CARTESIAN USE_NL USE_NL_WITH_INDEX
        USE_PARTITION_WISE_DISTINCT USE_PARTITION_WISE_GBY USE_PARTITION_WISE_WIF
        USE_PRIVATE_OUTLINES USE_SCALABLE_GBY_INVDIST USE_SEMI USE_STORED_OUTLINES
        USE_TTT_FOR_GSETS USE_VECTOR_AGGREGATION USE_WEAK_NAME_RESL USING USING_NO_EXPAND
        UTF16BE UTF16LE UTF32 UTF8 V1 V2 VALIDATE VALIDATE_CONVERSION VALIDATION VALID_TIME_END
        VALUE VARIANCE VARRAY VARRAYS VAR_POP VAR_SAMP VECTOR VECTOR_ENCODE VECTOR_READ
        VECTOR_READ_TRACE VECTOR_TRANSFORM VECTOR_TRANSFORM_DIMS VECTOR_TRANSFORM_FACT
        VERIFIER VERIFY VERSION VERSIONING VERSIONS VERSIONS_ENDSCN VERSIONS_ENDTIME
        VERSIONS_OPERATION VERSIONS_STARTSCN VERSIONS_STARTTIME VERSIONS_XID VIEWS
        VIOLATION VIRTUAL VISIBILITY VISIBLE VOLUME VSIZE WAIT WALLET WEEK WEEKS WELLFORMED
        WHEN WHENEVER WHITESPACE WIDTH_BUCKET WINDOW WITHIN WITHOUT WITH_EXPRESSION
        WITH_PLSQL WORK WRAPPED WRAPPER WRITE XDB_FASTPATH_INSERT XID XML XML2OBJECT
        XMLATTRIBUTES XMLCAST XMLCDATA XMLCOLATTVAL XMLCOMMENT XMLCONCAT XMLDIFF XMLELEMENT
        XMLEXISTS XMLEXISTS2 XMLFOREST XMLINDEX_REWRITE XMLINDEX_REWRITE_IN_SELECT
        XMLINDEX_SEL_IDX_TBL XMLISNODE XMLISVALID XMLNAMESPACES XMLPARSE XMLPATCH
        XMLPI XMLQUERY XMLQUERYVAL XMLROOT XMLSCHEMA XMLSERIALIZE XMLTABLE XMLTOJSON
        XMLTOKENSET XMLTRANSFORM XMLTRANSFORMBLOB XMLTSET_DML_ENABLE XML_DIAG XML_DML_RWT_STMT
        XPATHTABLE XS XS_SYS_CONTEXT X_DYN_PRUNE YEARS YES ZONEMAP
        ))
      end
      def self.keywords_func
        @keywords_func ||= Set.new(%w(
        ABS ACOS ADD_MONTHS APPROX_COUNT APPROX_COUNT_DISTINCT APPROX_COUNT_DISTINCT_AGG
        APPROX_COUNT_DISTINCT_DETAIL APPROX_MEDIAN APPROX_PERCENTILE APPROX_PERCENTILE_AGG
        APPROX_PERCENTILE_DETAIL APPROX_RANK APPROX_SUM ASCII ASCIISTR ASIN ATAN ATAN2
        AVG BFILENAME BIN_TO_NUM BITAND CARDINALITY CAST CEIL CHARTOROWID CHR CLUSTER_DETAILS
        CLUSTER_DISTANCE CLUSTER_ID CLUSTER_PROBABILITY CLUSTER_SET COALESCE COLLATION
        COLLECT COMPOSE CONCAT CONVERT CON_DBID_TO_ID CON_GUID_TO_ID CON_NAME_TO_ID
        CON_UID_TO_ID CORR COS COSH COUNT COVAR_POP COVAR_SAMP CUME_DIST CURRENT_DATE
        CURRENT_TIMESTAMP CV DATAOBJ_TO_MAT_PARTITION DATAOBJ_TO_PARTITION DBTIMEZONE
        DECODE DECOMPOSE DENSE_RANK DEPTH DEREF DUMP EMPTY_BLOB EMPTY_CLOB EXISTSNODE
        EXP EXTRACT EXTRACTVALUE FEATURE_COMPARE FEATURE_DETAILS FEATURE_ID FEATURE_SET
        FEATURE_VALUE FIRST FIRST_VALUE FLOOR FROM_TZ GREATEST GROUPING GROUPING_ID
        GROUP_ID HEXTORAW INITCAP INSTR ITERATION_NUMBER JSON_ARRAY JSON_ARRAYAGG
        JSON_OBJECT JSON_OBJECTAGG JSON_QUERY JSON_TABLE JSON_VALUE LAG LAST LAST_DAY
        LAST_VALUE LEAD LEAST LENGTH LISTAGG LN LNNVL LOCALTIMESTAMP LOG LOWER LPAD
        LTRIM MAKE_REF MAX MEDIAN MIN MOD MONTHS_BETWEEN NANVL NCHR NEW_TIME NEXT_DAY
        NLSSORT NLS_CHARSET_DECL_LEN NLS_CHARSET_ID NLS_CHARSET_NAME NLS_COLLATION_ID
        NLS_COLLATION_NAME NLS_INITCAP NLS_LOWER NLS_UPPER NTH_VALUE NTILE NULLIF
        NUMTODSINTERVAL NUMTOYMINTERVAL NVL NVL2 ORA_DM_PARTITION_NAME ORA_DST_AFFECTED
        ORA_DST_CONVERT ORA_DST_ERROR ORA_HASH ORA_INVOKING_USER ORA_INVOKING_USERID
        PATH PERCENTILE_CONT PERCENTILE_DISC PERCENT_RANK POWER POWERMULTISET POWERMULTISET_BY_CARDINALITY
        PREDICTION PREDICTION_BOUNDS PREDICTION_COST PREDICTION_DETAILS PREDICTION_PROBABILITY
        PREDICTION_SET PRESENTNNV PRESENTV PREVIOUS RANK RATIO_TO_REPORT RAWTOHEX
        RAWTONHEX REFTOHEX REGEXP_COUNT REGEXP_INSTR REGEXP_REPLACE REGEXP_SUBSTR
        REMAINDER REPLACE ROUND ROUND ROWIDTOCHAR ROWIDTONCHAR ROW_NUMBER RPAD RTRIM
        SCN_TO_TIMESTAMP SESSIONTIMEZONE SET SIGN SIN SINH SOUNDEX SQRT STANDARD_HASH
        STATS_BINOMIAL_TEST STATS_CROSSTAB STATS_F_TEST STATS_KS_TEST STATS_MODE STATS_MW_TEST
        STATS_ONE_WAY_ANOVA STATS_WSR_TEST STDDEV STDDEV_POP STDDEV_SAMP SUBSTR SUM
        SYSDATE SYSTIMESTAMP SYS_CONNECT_BY_PATH SYS_CONTEXT SYS_DBURIGEN SYS_EXTRACT_UTC
        SYS_GUID SYS_OP_ZONE_ID SYS_TYPEID SYS_XMLAGG SYS_XMLGEN TAN TANH TIMESTAMP_TO_SCN
        TO_APPROX_COUNT_DISTINCT TO_APPROX_PERCENTILE TO_BINARY_DOUBLE TO_BINARY_FLOAT
        TO_BLOB TO_CHAR TO_CLOB TO_DATE TO_DSINTERVAL TO_LOB TO_MULTI_BYTE TO_NCHAR
        TO_NCLOB TO_NUMBER TO_SINGLE_BYTE TO_TIMESTAMP TO_TIMESTAMP_TZ TO_YMINTERVAL
        TRANSLATE TREAT TRIM TRUNC TZ_OFFSET UID UNISTR UPPER USER USERENV VALIDATE_CONVERSION
        VALUE VARIANCE VAR_POP VAR_SAMP VSIZE WIDTH_BUCKET XMLAGG XMLCAST XMLCDATA
        XMLCOLATTVAL XMLCOMMENT XMLCONCAT XMLDIFF XMLELEMENT XMLEXISTS XMLFOREST XMLISVALID
        XMLPARSE XMLPATCH XMLPI XMLQUERY XMLROOT XMLSEQUENCE XMLSERIALIZE XMLTABLE
        XMLTRANSFORM
        ))
      end

      def self.keywords_type
        # sources:
        @keywords_type ||= Set.new(%w(
        CHAR BYTE VARCHAR2 NCHAR NVARCHAR2
        NUMBER FLOAT BINARY_FLOAT BINARY_DOUBLE
        LONG RAW
        DATE TIMESTAMP INTERVAL LOCAL TIME ZONE TO MONTH SECOND YEAR DAY
        BLOB CLOB NCLOB BFILE
        UROWID
        CHARACTER VARYING VARCHAR NATIONAL CHARACTER
        NUMERIC DECIMAL DEC INTEGER INT SMALLINT
        FLOAT DOUBLE PRECISION REAL
        SDO_GEOMETRY SDO_TOPO_GEOMETRY SDO_GEORASTER
        REF ANYTYPE ANYDATA ANYDATASET XMLTYPE HTTPURITYPE XDBURITYPE DUBRITYPE
        BOOLEAN PLS_INTEGER BINARY_INTEGER SIMPLE_FLOAT SIMPLE_INTEGER SIMPLE_DOUBLE SYS_REFCURSOR
        ))
      end


      state :root do
        delimiter_map = { '{' => '}', '[' => ']', '(' => ')', '<' => '>' }
        rule %r/\s+/m, Text
        rule %r/--.*/, Comment::Single
        rule %r(/\*), Comment::Multiline, :multiline_comments
        rule %r/q'(.)/i  do |m|
            #open = Regexp.escape(m[1])
            close = Regexp.escape(delimiter_map[m[1]] || m[1])
            # the opening q'X
            token Operator
            push do
                rule %r/(?:#{close}[^']|[^#{close}]'|[^#{close}'])+/m, Str::Other
                rule %r/#{close}'/, Operator, :pop!
            end
        end
        rule %r/'/, Operator, :single_string
        #
        # A double-quoted string refers to a database object in our default SQL
        rule %r/"/, Operator, :double_string

        ### we do not use backticks in Oracle. I do not know the rules for other sql engines
        ###rule %r/`/, Name::Variable, :backtick
        
        rule %r/[+-]?(?:(?:\.\d+(?:[eE][+-]?\d+)?)|\d+\.(?:\d+(?:[eE][+-]?\d+)?)?)/, Num::Float
        rule %r/[+-]?\d+/, Num::Integer
        
        rule %r/%(?:TYPE|ROWTYPE)\b/i, Name::Attribute

        # longer ones come first on purpose!
        rule %r/=>|\|\||\*\*|<<|>>|\.\.|<>|[:!~^<>]=|[-+%\/*=<>@&!^\[\]]/, Operator
        rule %r/(NOT|AND|OR|LIKE|BETWEEN|IN)(\s)/im do
            groups Operator, Text
        end
        rule %r/(IS)(\s+)(?:(NOT)(\s+))?(NULL\b)/im do
            groups Operator, Text, Operator, Text, Operator
        end

        rule %r/[;:()\[\],.]/, Punctuation

        # this madness is to keep the word "replace" from being treated as a builtin function in this context
        rule %r/(?:(replace)(\s+))?(package|function|procedure|type)(?:(\s+)(body))?(\s+)(\w[\w\d\$]*)/im do
            groups Keyword::Reserved, Text, Keyword::Reserved, Text, Keyword::Reserved, Text, Name
        end

        rule %r/(\$(?:IF|THEN|ELSE|ELSIF|ERROR|END|(?:\$\w+)))(\s+)/im do
            groups Comment::Preproc, Text
        end

        rule %r/\w[\w\d\$]*/ do |m|
          if self.class.keywords_type.include? m[0].upcase
            token Keyword::Type 
            #Name::Builtin
          elsif self.class.keywords_func.include? m[0].upcase
            token Name::Function
          elsif self.class.keywords.include? m[0].upcase
            token Keyword::Reserved
          elsif self.class.keywords_nresvd.include? m[0].upcase
            token Keyword
          else
            token Name
          end
        end

      end

      state :multiline_comments do
        rule %r/([*][^\/]|[^*])+/m, Comment::Multiline
        rule %r([*]\/), Comment::Multiline, :pop!
      end


      #state :backtick do
      #  rule %r/\\./, Str::Escape
      #  rule %r/``/, Str::Escape
      #  rule %r/`/, Name::Variable, :pop!
      #  rule %r/[^\\`]+/, Name::Variable
      #end

      state :single_string do
        rule %r/\\./, Str::Escape
        rule %r/''/, Str::Escape
        rule %r/'/, Operator, :pop!
        rule %r/[^\\']+/m, Str::Single
      end

      state :double_string do
        rule %r/\\./, Str::Escape
        rule %r/""/, Str::Escape
        rule %r/"/, Operator, :pop!
        rule %r/[^\\"]+/m, Name::Variable
      end


    end
  end
end

Staging Tables vs Single SQL

Sun, 06 Mar 2022 10:30:00 -0500

Introduction

A batch technique I often encounter creates staging tables for intermediate results, then breaks a problem down into multiple smaller steps, each one populating (and sometimes updating!) staging tables before consolidating into a final result. The argument is that breaking down the problem into smaller, more easily understood steps is a good programming practice, which on its face is true. There is also an advantage in being able to inspect the intermediate results while doing development and QA (and perhaps in production support).

Small Steps Using Staging Tables

Assuming the developer’s code is optimized as well as the Oracle optimizer would if given the entire problem, the only extra cost is the instantiation of intermediate results. Generally, I have not found the assumption to be true, but let’s give practitioners of this pattern the benefit of the doubt. Most of the time one can achieve reasonable performance using this technique.

Yet the overhead in undo/redo logging for these staging tables is not free. The total run-time will be longer using this technique, sometimes substantially. The DBA is probably gathering statistics in a job that will process your staging tables needlessly. In addition, we are bloating the undo and redo logs, plus these tables get backed up and take up room on “disk”.

If you would like a refresher course on undo and redo, my personal favorite is from Tom Kyte’s book Expert Oracle Architecture.

For our purposes just know those staging tables are not free, even if you use direct path load.

Furthermore, practitioners often employ UPDATEs and DELETEs which are much more expensive in Oracle than INSERTs. If you can gather the information you need while doing the INSERT, it is difficult to justify doing so after the fact with an UPDATE even to simplify your understanding of the solution. If you find yourself adding indexes to your staging table to facilitate subsequent operations, you might want to rethink your approach. (Of course, there are exceptions where the cost of creating an index on a staging table and using it to perform updates/deletes is the best answer, but it is extremely rare.)

Using WITH Subquery Factoring Alternative

Multiple Small Steps

We can write the same task in a single SQL statement using the WITH syntax to create multiple sequential views, aka Common Table Expressions (CTE). This retains the advantage of breaking the problem into multiple smaller steps while not paying the undo/redo cost of the staging tables. Instead, the execution plan allows the database engine to merrily pipeline results from join to join without ever writing out the intermediate resultsets. (Well, technically it can buffer intermediate resultsets in the temp tablespace between operations, and some of that may write to disk if there is not enough memory, but from a logical standpoint nothing is written to the database).

Not only that, but the optimizer can merge the views into the main query when it finds a better plan than the one you envisioned when you designed the program and broke it down into parts. Letting Oracle do the entire set of work in a single statement is almost always the most efficient and fastest solution.

From the developer’s point of view, we have broken down the problem into smaller, understandable chunks as CTE’s. It is the Oracle optimizer that is taking on the complexity, not the developer. Granted, your explain plan is going to be bigger and take some effort to relate to your code, but it should be in the wheelhouse for any serious Oracle practitioner to understand what the Optimizer does with your query.

Intermediate Resultsets

As for being able to see intermediate resultsets, at least during development and performance testing you can easily alter the query to stop after any given CTE, then select from that CTE. You can select the entire resultset or use a where clause or even analytic. For example:

WITH step1 AS (
    ...
), step2 AS (
    ...
), step3 AS (
    ...
) SELECT *
FROM step 3
INNER JOIN step2
    ON ...

You make a copy and interject your debug select like so:

WITH step1 AS (
    ...
), step2 AS (
    ...

) select mycollist from step2 where xyz = ‘mykey’;
– remainder of quuery does not run

), step3 AS (
    ...
) SELECT *
FROM step 3
INNER JOIN step2
    ON ...

It can also be helpful to do this if you want to look at the Explain Plan for the smaller subset of the query.

When you want to resume going through the query and looking at parts, you can put dash-dash in front of that line and continue going down the query testing each part. That way you can come back to it for further analysis.

CTE Used Multiple Times

If you use a CTE within the full query more than once, then Oracle must instantiate it. This will appear as Temp Table Transformation with a Load As Select below it near the start of the Explain Plan. For all practical purposes this is the same as populating a global temporary table. It uses the default temporary tablespace. As far as I can tell, this is effectively a direct path load of the transient global temporary table that does not generate much undo (and by default creating undo also generates redo for that undo). This addresses another stated purpose for using staging tables – to be able to use them more than once.

In this scenario when you have a large resultset, it is still going to be almost as expensive to instantiate it in Temporary space as it would be to do a direct path load into a staging table (except for the redo logging). The only way around that is to figure out how to avoid reusing the CTE. You may surprise yourself and find that you can.

Cardinality and Statistics

Using WITH/CTE technique rather than staging tables, the optimizer has an advantage of knowing how each intermediate result was gathered. It can estimate the cardinality. When using staging tables, you must either gather statistics on the staging table or provide a cardinality hint to the using query. Gathering statistics can be even more expensive than the redo logging.

Discoveries from Refactoring Staging Tables to a Single SQL

NO_MERGE Hint

The optimizer will sometime merge subqueries together that you can prove via experimentation is a bad idea. Purists will tell you that you just don’t have your statistics right. I don’t argue. I just hint the damn thing.

For the example below, Oracle will almost always merge the view named keys into the main select and do a single HASH UNIQUE.

WITH keys AS (
    SELECT DISTINCT key1, key2, key3
    FROM key_source
) SELECT DISTINCT resultfieldlist
FROM keys k
INNER JOIN sourcetable s
    ON s.key1 = k.key1 AND s.key2 = k.key2 AND s.key3 = k.key3

I can prove that in case of the query I was tuning, we are better off gathering a DISTINCT set of keys before joining to sourcetable. The optimizer flat out refuses and merges the “set of keys” view into the join query so that it only has to do a single HASH UNIQUE after the join. Yet the duplication of the keys between those two sources is multiplicative. If we let Oracle have its way, the result set going into the single HASH UNIQUE the optimizer is so proud of is ginormous. I know from trial and error for this particular set of inputs, we are better off with two separate HASH UNIQUE operations.

In this scenario you can force Oracle to do two separate HASH UNIQUE operations by using the NO_MERGE hint:

WITH keys AS (
    SELECT /*+ NO_MERGE */ DISTINCT 
        key1, key2, key3
    FROM key_source
) SELECT DISTINCT resultfieldlist
FROM keys k
INNER JOIN sourcetable s
    ON s.key1 = k.key1 AND s.key2 = k.key2 AND s.key3 = k.key3

I’ve also seen cases where Oracle chooses to merge in a subquery containing an analytic, doing the join first and then the analytic and the filter using the analytic column that was supposed to be applied before doing the join.

WITH keys AS (
    SELECT ...
        ,ROW_NUMBER() OVER (PARTITION BY xyz ORDER by last_modified_dt DESC) AS rn
    FROM key_source
) SELECT resultfieldlist
FROM keys k
INNER JOIN sourcetable s
    ON s.key1 = k.key1 AND s.key2 = k.key2 AND s.key3 = k.key3
WHERE k.rn = 1

The optimizer has determined that it can reduce the resultset better by doing the join first, shrinking the number of rows from key_source, then doing the analytic sort and filter. Maybe the optimizer is right. When it is wrong, you will notice a humongous hash join taking forever in order to save a little bit on that sort. That is when you can try making it do the sort first using NO_MERGE:

WITH keys AS (
    SELECT /*+ NO_MERGE */ ...
        ,ROW_NUMBER() OVER (PARTITION BY xyz ORDER by last_modified_dt DESC) AS rn
    FROM key_source
) SELECT resultfieldlist
FROM keys k
INNER JOIN sourcetable s
    ON s.key1 = k.key1 AND s.key2 = k.key2 AND s.key3 = k.key3
WHERE k.rn = 1

I do not advise reaching for the NO_MERGE hint until you suspect a problem, and even then you should try it both ways. Need for the hint in these scenarios is the exception, as most of the time the optimizer is right. Yet the optimizer relies on rules of thumb and guesses that do not necessarily match your data. Even with good statistics with histograms available, much of the optimizer logic is based on probabilities and rules of thumb. It is not always right. Figuring out how to give the optimizer enough information to come up with the right answer, as one school of purists advocate, is not practical.

Fat Resultset and Big Joins

When we get into very large joins when one or both of the input datasets are very fat (lots of columns/big fat records), there is a technique from the staging table school of thought you might want to employ. The technique is to grab only the join keys from one of your sources (likely your target record that you are joining additional fields into) and use those keys to gather the data from the other table, then once you have the values you need, join them back into the main query.

Why do the join twice? Consider that the table you are reading is very large with many more rows than the one you are joining into. The proper way to do this is hash the one with less rows and probe it from the new table; yet, because it is so fat, hashing the entire thing overflows our available PGA memory pushing us off into a onepass or multipass hash join. In this scenario we could be better off just hashing the distinct set of join keys while we probe it from the large source table until we have only the data we need.

Eventually we still have to join to our big fat target resultset, but now we have a smaller footprint of data to do it with. We can reverse which resultset is hashed to the one that fits in memory.

I still am likely to use WITH subqueries to do it but have at times found a separate staging table is a faster answer. I have not completely grasped the reason, but suspect it revolves around the PGA size and the amount of memory paged to disk for our reuse of our main resultset.

Sometimes an Update Really is Faster

We have a very large and fat resultest, and need to update values in a very small number of rows. We need to make a full table scan pass through our data to select candidate rows (or the keys from candidate rows) and join them to our source. We cannot avoid that first full scan. If we have a staging table, we can bring the ROWID along, then use that for the MERGE ON clause join back into the staging table.

MERGE INTO staging_table t
USING (
    WITH lkup_value AS (
        SELECT s.rowid AS s_rowid, lk.update_value
            ROW_NUMBER() OVER (PARTITION BY s.key1, s.key2 ORDER BY lk.effective_date DESC) AS rn
        FROM staging_table s
        INNER JOIN lookup_table lk
            ON lk.key1 = s.key1 AND lk.key2 = s.key2...
                AND s.create_dt >= lk.effective_date
    ) SELECT s_rowid, update_value
    FROM lkup_value
    WHERE rn = 1
) q
ON (t.rowid = q.s_rowid)
WHEN MATCHED THEN UPDATE SET
    update_value = q.update_value

Consider how we would do this in a series of steps using WITH clause CTEs in a single INSERT without the staging table:

INSERT INTO staging_table
WITH x AS (
    ...
), lkup_value_1 AS (
        SELECT x.key1, x.key2, lk.update_value
            ROW_NUMBER() OVER (PARTITION BY x.key1, x.key2 ORDER BY lk.effective_date DESC) AS rn
        FROM x
        INNER JOIN lookup_table lk
            ON lk.key1 = x.key1 AND lk.key2 = x.key2...
                AND x.create_dt >= lk.effective_date
), lkup_value AS (
    SELECT key1, key2, update_value
    FROM lkup_value_1
    WHERE rn = 1
), y AS (
    SELECT x.field1, x.field2...
        ,CASE WHEN lk.key1 IS NOT NULL THEN lk.update_value ELSE x.update_value END AS update_value
    FROM x
    LEFT OUTER JOIN lkup_value lk
        ON lk.key1 = x.key1 AND lk.key2 = x.key2
)...

Even though we are only changing a few rows with clause ‘y’, we must run all of the rows from ‘x’ through this join with lkup_value clause. I have observed situations where keeping a staging table and doing the MERGE as shown above is superior to doing everything in a single query. This surprised me. I started the task with the expectation that I could eliminate all of the staging tables. It was not the answer I wanted, but it was the answer I found.

Conclusion

To break a database problem into small, understandable chunks you can use the WITH clause Common Table Expressions (CTE) instead of staging tables. There are more benefits than downside to this technique, but there are some exceptional circumstances where using intermediate staging tables is the best approach. Understand how the optimizer processes what you have designed as sequential steps (most of which are pipelined), and use hints when absolutely necessary to guide the optimizer away from suboptimal plans. A detailed understanding of the memory involved with HASH joins can help you design non-obvious solutions for very large dataset operations.

Inline External Tables for CSV Load

Mon, 24 Jan 2022 10:30:00 -0500

Yet Another CSV Load Option - Inline External Tables

While reviewing the Oracle What’s New documentation for 18c I found Private Temporary Tables which I wrote about in my last post on the neverending saga of loading CSV data. Also found in What’s New for 18c is Inline External Tables.

Just like with PTTs, Inline External Tables are DDL without the downside. It is ad-hoc, and can be dynamically generated. In order to use it you must have an Oracle DIRECTORY object granted to you with both READ and WRITE, plus EXECUTE privilege on DBMS_LOB (and UTL_FILE if you want to clean up after yourself).

Example

From our last post we had an example of CSV data with a header row that looked as such:

Employee ID, Last Name, First Name, nickname
999, "Baggins", "Bilbo", "badboy, ringbearer"
998, "Baggins", "Frodo",
997, "Orc", "Ogg", "i kill you"

We could get fancy and try to parse the header row like we did last time, but for this effort I’m going the cheap route and assume you, the developer who wants to load the CSV data, will hand craft the code.

Create File on Oracle Server

BEGIN
    DBMS_LOB.clob2file(q'[999, "Baggins", "Bilbo", "badboy, ringbearer"
998, "Baggins", "Frodo",
997, "Orc", "Ogg", "i kill you"]'
                        ,'TMP_DIR'
                        ,'temp_csv_load.csv'
    );
END;
/

Read from the Inline External Table

Now we read from the file:

    SELECT *
    FROM EXTERNAL(
        (
            "Employee ID"   VARCHAR2(4000)
            ,"Last Name"    VARCHAR2(4000)
            ,"First Name"   VARCHAR2(4000)
            ,"nickname"     VARCHAR2(4000)
        )
        TYPE ORACLE_LOADER
        DEFAULT DIRECTORY TMP_DIR
        ACCESS PARAMETERS (
            RECORDS DELIMITED BY NEWLINE
            FIELDS TERMINATED BY ',' 
            OPTIONALLY ENCLOSED BY '"'
            MISSING FIELD VALUES ARE NULL
        )
        LOCATION ('temp_csv_load.csv')
        REJECT LIMIT UNLIMITED
    ) temp_csv_load_ext
    ;

The export from SQL Developer is adding the double quotes here. These are not in the data.

"Employee ID"                 "Last Name"                   "First Name"                  "nickname"                    
"999"                         "Baggins"                     "Bilbo"                       "badboy, ringbearer"          
"998"                         "Baggins"                     "Frodo"                       ""                            
"997"                         "Orc"                         "Ogg"                         "i kill you"                  

You could of course have the external table sqlldr driver convert to dates and numbers as needed. As far as I can tell you have everything at your disposal that is there for a normal external table.

The line

OPTIONALLY ENCLOSED BY '"'

makes sqlldr parse CSV data in a way that I believe mostly comports with the RFC on CSV data. It will handle most of the test cases I threw at it and you are unlikely to give it the oddball stuff.

The line

MISSING FIELD VALUES ARE NULL

is because we do not have a trailing comma after the last field (which would be delimited data rather than separated) and because the sqlldr syntax of ‘TRAILING NULLCOLS’ I would pick is not available in the external table driver. Maddening. Without it our second record would fail to load.

Cleanup

Being a good citizen, we remove our trash:

BEGIN
    UTL_FILE.fremove('TMP_DIR','temp_csv_load.csv');
END;
/

Drawback

The big drawback to this technique (aside from how hard it is to get the external table definition right) is that it requires you have READ/WRITE privs on a directory on the database server. In many organizations this is forbidden. I retrofitted forty something load jobs from external table to sqlldr because of that restriction imposed upon us by architecture/DBA teams a few years back. We can debate whether that is reasonable or not, but it is what it is.

Conclusion

There are plenty of client tools that will generate load data for you from CSV files or even directly from Excel. As nice as those are, you probably cannot use them for Continuous Improvement deployments. I’m trying to come up with a way to make that better. This is one more technique we might be able to use.

CSV Clob and Private Temporary Table

Sun, 23 Jan 2022 10:30:00 -0500

Deploying Table Data Using CSV

I have been obsessing about parsing CSV data with a focus on the use case of code promotion through Continuous Improvement/Devops process. The current method of deploying a bunch of single INSERT statements rubs me the wrong way. It is ridiculous.

I get the feeling I’m the only one who cares, but hey, I’m the one who matters!

In my last post on the topic I described how one could use some new tools I created in perlish_util_udt to parse a CSV clob into records and fields, then use that parsed data in a SQL statement, perhaps including DML.

That blog post left it still a bit rough though. The user would need to remember or relearn the syntax for two related functions to do the parsing, and the syntax for using an object method in a SQL query is easy to forget about or mess up.

I had also been experimenting with Polymorphic Table Functions for this purpose because you can determine the resultset structure at run time, which is important for this task. Yet PTFs are complicated. Maybe too complicated.

I was rereading the “What’s New” Oracle documentation for the last several releases and Private Temporary Tables (added in 18c) caught my eye. Much of the issue we face with this use case is that it needs to be dynamic and DDL is not dynamic. It is resource expensive and regulated in most production environments. Private Temporary Tables are DDL without the downside. It does not alter the data dictionary (well, maybe there is something going on as it take advantage of the TEMPORARY tablespace), it does not leave anything behind after a session completes, and it is not regulated by our corporate rules.

Private Temporary Tables let us define our column list on the fly!

perlish_util_udt.create_ptt_csv

This static procedure recently added to perlish_util_udt combines the operations demonstrated in my prior blog post about CSV data to build and populate a Private Temporary Table to contain your data from the CSV clob. How cool is that?

Example

BEGIN
    perlish_util_udt.create_ptt_csv('Employee ID, Last Name, First Name, nickname
999, "Baggins", "Bilbo", "badboy, ringbearer"
998, "Baggins", "Frodo",
997, "Orc", "Ogg", "i kill you"');
END;
/
select "Employee ID", "Last Name", "First Name", "nickname"
FROM ora$ptt_csv
;

"Employee ID"                 "Last Name"                   "First Name"                  "nickname"                    
"999"                         "Baggins"                     "Bilbo"                       "badboy, ringbearer"          
"998"                         "Baggins"                     "Frodo"                       ""                            
"997"                         "Orc"                         "Ogg"                         "i kill you"                  

The value of “nickname” in the Fodo record is NULL.

That is a lot easier than doing the parsing into lines and the parsing into fields and selecting by index number we were using before. Under the covers that is what it does, but now we have something that is easy to use.

The Code

Given the prior work with perlish_util_udt.split_clobs_to_lines and perlish_util_udt.split_csv, this was not difficult to implement.

	STATIC PROCEDURE create_ptt_csv (
         -- creates private temporary table ora$ptt_csv with columns named in first row of data case preserved.
         -- All fields are varchar2(4000)
	     p_clob         CLOB
	    ,p_separator    VARCHAR2    DEFAULT ','
	    ,p_strip_dquote VARCHAR2    DEFAULT 'Y' -- also unquotes \" and "" pairs within the field to just "
	) IS
        v_rows      arr_varchar2_udt := perlish_util_udt.split_clob_to_lines(p_clob);
        v_cols      perlish_util_udt;

The first step, which is in the variable declarations, splits the clob up into an array of rows. We are going to parse the first row into v_cols which we make into an object type so we can take advantage of our map and join methods to build the SQL we need to execute.

        v_sql       CLOB;
    BEGIN
        v_cols := perlish_util_udt(perlish_util_udt.split_csv(v_rows(1), p_separator => p_separator, p_strip_dquote => 'Y'));
        -- remove the header row so can bind the array to read the data
        v_rows.DELETE(1);

We parse the first row differently than the rest. We are going to enclose the data in double quotes so we must strip them if they exist. After we get our data from the first row, we delete it from the collection. We will later bind the collection array to a SQL statement and do not want the first row to be present.

        v_sql := 'DROP TABLE ora$ptt_csv';
        BEGIN
            EXECUTE IMMEDIATE v_sql;
        EXCEPTION WHEN OTHERS THEN NULL;
        END;

Drop the PTT if it already exists.

        v_sql := 'CREATE PRIVATE TEMPORARY TABLE ora$ptt_csv(
'
            ||v_cols.map('"$_"    VARCHAR2(4000)').join('
,')
            ||'
)'
            ;
        DBMS_OUTPUT.put_line(v_sql);
        EXECUTE IMMEDIATE v_sql;

We use our collection of column names from the first row to create the column defintions for the PTT. map and join are handy for this. As you can see we do not try to figure out what kind of data each column is. We just stuff each column value into a VARCHAR2(4000) field.

The serveroutput of this code from running the example above was:

CREATE PRIVATE TEMPORARY TABLE ora$ptt_csv(
"Employee ID"    VARCHAR2(4000)
,"Last Name"    VARCHAR2(4000)
,"First Name"    VARCHAR2(4000)
,"nickname"    VARCHAR2(4000)
)

Now for the INSERT.

        v_sql := q'[INSERT INTO ora$ptt_csv 
WITH a AS (
    SELECT perlish_util_udt(
            perlish_util_udt.split_csv(t.column_value, p_separator => :p_separator, p_strip_dquote => :p_strip_dquote, p_keep_nulls => 'Y')
        ) AS p
    FROM TABLE(:bind_array) t
) SELECT ]'
        ||v_cols.map('x.p.get($##index_val##) AS "$_"').join('
,')
        ||'
FROM a x';
        -- must use table alias and fully qualify object name with it to be able to call function or get attribute of object
        -- Thus alias x for a.
        DBMS_OUTPUT.put_line(v_sql);
        EXECUTE IMMEDIATE v_sql USING p_separator, p_strip_dquote, v_rows;

    END -- crate_ptt_csv
    ;

Building the INSERT statement was a little tricky because we need to use the object function get with the array index value for each column. Once again map and join are handing for building this. I added the ‘$##index_val##’ substitution to map for this release.

The serveroutput of this section from the example:

INSERT INTO ora$ptt_csv
WITH a AS (
    SELECT perlish_util_udt(
            perlish_util_udt.split_csv(t.column_value, p_separator => :p_separator, p_strip_dquote => :p_strip_dquote, p_keep_nulls => 'Y')
        ) AS p
    FROM TABLE(:bind_array) t
) SELECT x.p.get(1) AS "Employee ID"
,x.p.get(2) AS "Last Name"
,x.p.get(3) AS "First Name"
,x.p.get(4) AS "nickname"
FROM a x

And of course you have seen the results. I’m starting to like this. Sure, I’ve been seduced by the dark side and am writing PL/SQL code that looks like Perl, but I am what I am.

Protecting/Hiding Data in Oracle

Sat, 22 Jan 2022 10:30:00 -0500

The Ask

While experimenting with as_sftp, a package by Anton Scheffer that implements a Secure File Transfer Protocol (SFTP) utility in Oracle PL/SQL, I encountered an issue (or want) of storing SSH private keys in the database. That opens a can of worms because private keys should be secured as much as possible.

On a Unix system the private keys are kept in files in ~/.ssh directory with permissions set to be readable only by the user login associated with that HOME directory. Of course anyone with root or sudo privs can get the file, but that is the level of security we have come to expect for SSH client implementations. It is also one of the reasons some corporations have issues with ssh and go to lengths to prevent using it except where absolutely necessary. I digress.

I need to store private keys in the database with the following requirements:

Not visible to any user except the schema owner (and DBA, but only because we can’t keep them out).
Accessible to a particular procedure that calls the login method of as_sftp.
Not accessible to any other procedures or queries.
Not visible to the schema owner if possible. The reason for this odd seeming requirement is that in most corporate systems a schema owner for deployed code is a shared account. Even if it is protected from login, there are multiple developers who deploy code into the schema. We want to control access to the single deployer and the procedure that needs the private key.

Hiding Data Use Cases

I can see altering the use case to have separate privileges for the DML operations versus the SFTP Login, but I’m going to combine them for now.

As shown in the diagram, the authorized user does not have a use case for obtaining the private key directly or even to tell that there is a key available. That is by design.

Options

I considered a package with the source obfuscated via the utility wrap. From the documentation though

Although wrapping a compilation unit helps to hide the algorithm and makes reverse-engineering difficult, Oracle Corporation does not recommend it as a secure method for hiding passwords or table names. Obfuscating a PL/SQL unit prevents most users from examining the source code, but might not stop all attempts.

It might be good enough for our use case, but there are other issues. This package or function will be executable by the schema owner, who in our target environment is a shared account. We also do not control all database security and a surprising number of other accounts may have elevated privliges including EXECUTE ANY PROCEDURE (yea, don’t. OK. Not my circus. Not my monkeys.).

As for storing the key in a table, even more frequently found in the wild is the grant of SELECT ANY TABLE.

I considered putting some sort of encryption on the text of the key in the table, but that just kicks the can around because now the key for that encryption needs to be stored somewhere. Not helpful.

I briefly flirted with using a Context, but it has to be loaded somehow and we are back to protecting that operation.

I settled on Oracle Fine-Grained Access Control (See Oracle Database Security Guide and DBMS_RLS for your release).

The solution won’t stop that person with EXECUTE ANY PROCEDURE from running our login procedure which uses our private key (shame on the DBA!), but it prevents them from grabbing the private key and using it elsewhere. Even this issue might be solved if we add a check for calling user in the policy, but I have not gone there yet.

There is an extensive amount of debug logging available in as_sftp. I have not taken the time to explore it all to see if the private keys can be exposed.

I am not a security expert. It is possible there are more glaring holes in what I have built. Everything I’ve checked suggests this is good against any but a SYSDBA privileged user account, yet I’m not going to claim this is a secure solution. I just do not know. This is an exercise in implementing Fine Grained Access Control.

Fine Grained Access Control Configuration

The documentation and examples center around use of a context that you populate with a login trigger to restrict access to particular logins or scenarios as to how the session was entered. Much of that does not fit my use case.

What does fit is that we can install functions that control whether or not a SELECT, INSERT, UPDATE or DELETE operation will or will not work based on logic we implement. In this context “not work” means a SELECT will return 0 rows but no error, while a DML operation will return either 0 rows updated/deleted, or in the case of insert, a policy error exception. No user or procedure can manipulate the data in this table without passing our policy check.

The exceptions to this are that the schema owner can perform DDL on the table including TRUNCATE and DROP, and anyone with SYSDBA priv can do whatever they want. These conditions are acceptable to me. To take it any further we would need to create a separate schema for the table and packages. That might be best, but in most corporate environments, it is expensive to do. That said, if I was installing as_sftp as a DBA for general use that is where I would go with it.

My intent is for the policy check functions to look at the call stack and only allow the operation to succeed if we are called by a specific schema owned package function/procedure. Mostly that means my supplemental login procedure that retrieves the private key and calls as_sftp.login.

Implementation

This implementation is for a single schema owner. If it was to be shared across multiple schemas, there would likely need to be an additional primary key field in the table for who is running the operation. That would also need to consider how Mr. Scheffer configured the as_sfpt_known_hosts operation. It may be possible to define this with INVOKER rights and force each schema owner to implement their own tables, but that seems draconian. Tracking everything by the user identification of the caller would be my choice.

as_sftp_private_keys Table

CREATE TABLE as_sftp_private_keys
(
    host  VARCHAR2(1000) NOT NULL
    ,id   VARCHAR2(128) NOT NULL
    ,key  CLOB NOT NULL
    ,CONSTRAINT as_sftp_private_keys_pk PRIMARY KEY(host, id)
);

as_sftp_keymgmt Package Specfication

CREATE OR REPLACE PACKAGE as_sftp_keymgmt AS
    --
    -- Important! The private key lookup is case sensitive on i_host and i_user.
    --
    PROCEDURE login(i_host VARCHAR2, i_user VARCHAR2, i_passphrase VARCHAR2 := NULL, i_log_level pls_integer := null);
-- comment out this function when done testing. It should not be public
    --FUNCTION get_priv_key(i_host VARCHAR2, i_user VARCHAR2) RETURN CLOB;
    --
    -- When keymgmt_security is activated (fine grained access control)
    -- These three methods are the only way to manipuate the data in the table as_sftp_private_keys
    -- other than to truncate it or do the task as sysdba.
    -- You cannot read the data at all as get_priv_key is a private function that only login() can call.
    --
    PROCEDURE insert_priv_key(i_host VARCHAR2, i_user VARCHAR2, i_key CLOB);
    PROCEDURE update_priv_key(i_host VARCHAR2, i_user VARCHAR2, i_key CLOB);
    PROCEDURE delete_priv_key(i_host VARCHAR2, i_user VARCHAR2);
END as_sftp_keymgmt;

While testing I made the function get_priv_key public so that I could validate what it was doing. It should be private.

The procuedure login will lookup the private key (using get_priv_key) and call as_sftp.login using it.

The three DML procedures allow operations on individual private key records for anyone granted execute on the package. As noted, it might be better to put those in a separate package.

So far what we have implemented will work without installing the fine grained access control components. Although anyone with SELECT priv on the table can see our keys, the storage of keys and use of them automatically during login will work much like SSH/SFTP does on Unix by looking up the key.

as_sftp_keymgmt_security PACKAGE Specification

as_sftp_keymgmt_security package implements the policy checks that will, when installed via DBMS_RLS, control whether access to table as_sftp_private_keys is permitted.

CREATE OR REPLACE PACKAGE as_sftp_keymgmt_security
AS
    --
    -- These control access to records in the table as_sftp_private_keys
    --
    FUNCTION user_data_select_security (owner VARCHAR2, objname VARCHAR2)
    RETURN VARCHAR2;
    FUNCTION user_data_insert_security (owner VARCHAR2, objname VARCHAR2)
    RETURN VARCHAR2;
    FUNCTION user_data_update_security (owner VARCHAR2, objname VARCHAR2)
    RETURN VARCHAR2;
    FUNCTION user_data_delete_security (owner VARCHAR2, objname VARCHAR2)
    RETURN VARCHAR2;
END as_sftp_keymgmt_security;

We will explore the package body after seeing how it is tied to the table.

install_keymgmt_security.sql

This install script associates the functions in package as_sftp_keymgmt_security with each of the operations ‘SELECT’, ‘INSERT’, ‘UPDATE’, ‘DELETE’ on table as_sftp_private_keys.

The package as_sftp_keymgmt_security must be both publicly executable and have a public synonym for this to work. If you do not have the privilege to create a public synonym, the DBA must do it for you.

You may also need the DBA to ‘GRANT EXECUTE ON DBMS_RLS TO your_schema;’, though it is common enough to grant it to ‘PUBLIC’.

Of note is that the add_policy for ‘INSERT’ uses an additional parameter because ‘INSERT’ statements do not have ‘WHERE’ clauses. The implementation checks your condition after the insert, then raises an EXCEPTION if the operation is not allowed. This is different than the others where it quietly adds a WHERE condition to cause your operation to not match any rows.

GRANT EXECUTE ON as_sftp_keymgmt_security TO public;
CREATE OR REPLACE PUBLIC SYNONYM as_sftp_keymgmt_security FOR lee.as_sftp_keymgmt_security ;
BEGIN
    BEGIN
        DBMS_RLS.drop_policy('LEE', 'AS_SFTP_PRIVATE_KEYS', 'USER_DATA_SELECT_POLICY');
    EXCEPTION WHEN OTHERS THEN NULL;
    END;
    DBMS_RLS.add_policy('LEE', 'AS_SFTP_PRIVATE_KEYS', 'USER_DATA_SELECT_POLICY',
                      'LEE', 'AS_SFTP_KEYMGMT_SECURITY.USER_DATA_SELECT_SECURITY',
                      'SELECT');
    BEGIN
        DBMS_RLS.drop_policy('LEE', 'AS_SFTP_PRIVATE_KEYS', 'USER_DATA_INSERT_POLICY');
    EXCEPTION WHEN OTHERS THEN NULL;
    END;
    DBMS_RLS.add_policy('LEE', 'AS_SFTP_PRIVATE_KEYS', 'USER_DATA_INSERT_POLICY',
                      'LEE', 'AS_SFTP_KEYMGMT_SECURITY.USER_DATA_INSERT_SECURITY',
                      'INSERT'
                    ,TRUE -- needed because insert does not have where clause. check condition after insert
                );
    BEGIN
        DBMS_RLS.drop_policy('LEE', 'AS_SFTP_PRIVATE_KEYS', 'USER_DATA_UPDATE_POLICY');
    EXCEPTION WHEN OTHERS THEN NULL;
    END;
    DBMS_RLS.add_policy('LEE', 'AS_SFTP_PRIVATE_KEYS', 'USER_DATA_UPDATE_POLICY',
                      'LEE', 'AS_SFTP_KEYMGMT_SECURITY.USER_DATA_UPDATE_SECURITY',
                      'UPDATE');
    BEGIN
        DBMS_RLS.drop_policy('LEE', 'AS_SFTP_PRIVATE_KEYS', 'USER_DATA_DELETE_POLICY');
    EXCEPTION WHEN OTHERS THEN NULL;
    END;
    DBMS_RLS.add_policy('LEE', 'AS_SFTP_PRIVATE_KEYS', 'USER_DATA_DELETE_POLICY',
                      'LEE', 'AS_SFTP_KEYMGMT_SECURITY.USER_DATA_DELETE_SECURITY',
                      'DELETE');
END;

as_sftp_keymgmt_security Package Body

Now we get to the fun stuff. How are we determining that ONLY a specific package and method are allowed access? We look at the call stack. It is tricky though because the DBMS_RLS operations insert themselves into the call stack! You must look back a few levels to find the original caller.

CREATE OR REPLACE PACKAGE BODY as_sftp_keymgmt_security IS
    FUNCTION user_data_select_security (owner VARCHAR2, objname VARCHAR2)
    RETURN VARCHAR2
    IS
        v_owner varchar2(1024);
        v_name varchar2(1024);
        v_lineno number;
        v_caller_t varchar2(1024);

        v_depth BINARY_INTEGER := UTL_CALL_STACK.dynamic_depth;
    BEGIN
        IF v_depth < 4 THEN
            DBMS_OUTPUT.put_line('security check found not called by LEE.AS_SFTP_KEYMGMT.GET_PRIV_KEY.');
            dbms_output.put_line('call stack less than 4: '||v_depth);
            DBMS_OUTPUT.put_line('Not allowing rows to be selected.');
            RETURN '1=0';
        END IF;
        /*
        FOR i in 1..v_depth
        LOOP
            v_owner := UTL_CALL_STACK.owner(i);
            v_name := UTL_CALL_STACK.concatenate_subprogram(UTL_CALL_STACK.subprogram(i));
            dbms_output.put_line('i='||i||' owner: '||v_owner||' name: '||v_name);
        END LOOP;
        */
        v_owner := UTL_CALL_STACK.owner(4);
        v_name := UTL_CALL_STACK.concatenate_subprogram(UTL_CALL_STACK.subprogram(4));
        --dbms_output.put_line('owner: '||v_owner||' name: '||v_name);
        --owa_util.who_called_me(v_owner, v_name, v_lineno, v_caller_t);
        --dbms_output.put_line('owner: '||v_owner||' name: '||v_name||' lineno: '||v_lineno||' caller_t: '||v_caller_t);
        --dbms_output.put_line(DBMS_UTILITY.format_call_stack);
        IF v_owner = 'LEE' AND v_name = 'AS_SFTP_KEYMGMT.GET_PRIV_KEY' THEN
            RETURN NULL;
        ELSE
            DBMS_OUTPUT.put_line('security check found not called by LEE.AS_SFTP_KEYMGMT.GET_PRIV_KEY.');
            DBMS_OUTPUT.put_line('was called by owner: '||v_owner||' name: '||v_name);
            DBMS_OUTPUT.put_line('Not allowing rows to be selected.');
            RETURN '1=0';
        END IF;
        --RETURN NULL;
    END user_data_select_security
    ;

    FUNCTION user_data_insert_security (owner VARCHAR2, objname VARCHAR2)
    RETURN VARCHAR2
    IS
        v_owner varchar2(1024);
        v_name varchar2(1024);
        v_lineno number;
        v_caller_t varchar2(1024);

        v_depth BINARY_INTEGER := UTL_CALL_STACK.dynamic_depth;
    BEGIN
        IF v_depth < 4 THEN
            DBMS_OUTPUT.put_line('security check found not called by LEE.AS_SFTP_KEYMGMT.INSERT_PRIV_KEY.');
            dbms_output.put_line('call stack less than 4: '||v_depth);
            DBMS_OUTPUT.put_line('Not allowing rows to be selected.');
            RETURN '1=0';
        END IF;
        /*
        FOR i in 1..v_depth
        LOOP
            v_owner := UTL_CALL_STACK.owner(i);
            v_name := UTL_CALL_STACK.concatenate_subprogram(UTL_CALL_STACK.subprogram(i));
            dbms_output.put_line('i='||i||' owner: '||v_owner||' name: '||v_name);
        END LOOP;
        */
        v_owner := UTL_CALL_STACK.owner(4);
        v_name := UTL_CALL_STACK.concatenate_subprogram(UTL_CALL_STACK.subprogram(4));
        --dbms_output.put_line('owner: '||v_owner||' name: '||v_name);
        --owa_util.who_called_me(v_owner, v_name, v_lineno, v_caller_t);
        --dbms_output.put_line('owner: '||v_owner||' name: '||v_name||' lineno: '||v_lineno||' caller_t: '||v_caller_t);
        --dbms_output.put_line(DBMS_UTILITY.format_call_stack);
        IF v_owner = 'LEE' AND v_name = 'AS_SFTP_KEYMGMT.INSERT_PRIV_KEY' THEN
            RETURN NULL;
        ELSE
            DBMS_OUTPUT.put_line('security check found not called by LEE.AS_SFTP_KEYMGMT.INSERT_PRIV_KEY.');
            DBMS_OUTPUT.put_line('was called by owner: '||v_owner||' name: '||v_name);
            DBMS_OUTPUT.put_line('Not allowing rows to be inserted.');
            RETURN '1=0';
        END IF;
        --RETURN NULL;
    END user_data_insert_security
    ;
    FUNCTION user_data_update_security (owner VARCHAR2, objname VARCHAR2)
    RETURN VARCHAR2
    IS
        v_owner varchar2(1024);
        v_name varchar2(1024);
        v_lineno number;
        v_caller_t varchar2(1024);

        v_depth BINARY_INTEGER := UTL_CALL_STACK.dynamic_depth;
    BEGIN
        IF v_depth < 4 THEN
            DBMS_OUTPUT.put_line('security check found not called by LEE.AS_SFTP_KEYMGMT.UPDATE_PRIV_KEY.');
            dbms_output.put_line('call stack less than 4: '||v_depth);
            DBMS_OUTPUT.put_line('Not allowing rows to be selected.');
            RETURN '1=0';
        END IF;
        /*
        FOR i in 1..v_depth
        LOOP
            v_owner := UTL_CALL_STACK.owner(i);
            v_name := UTL_CALL_STACK.concatenate_subprogram(UTL_CALL_STACK.subprogram(i));
            dbms_output.put_line('i='||i||' owner: '||v_owner||' name: '||v_name);
        END LOOP;
        */
        v_owner := UTL_CALL_STACK.owner(4);
        v_name := UTL_CALL_STACK.concatenate_subprogram(UTL_CALL_STACK.subprogram(4));
        --dbms_output.put_line('owner: '||v_owner||' name: '||v_name);
        --owa_util.who_called_me(v_owner, v_name, v_lineno, v_caller_t);
        --dbms_output.put_line('owner: '||v_owner||' name: '||v_name||' lineno: '||v_lineno||' caller_t: '||v_caller_t);
        --dbms_output.put_line(DBMS_UTILITY.format_call_stack);
        IF v_owner = 'LEE' AND v_name = 'AS_SFTP_KEYMGMT.UPDATE_PRIV_KEY' THEN
            RETURN NULL;
        ELSE
            DBMS_OUTPUT.put_line('security check found not called by LEE.AS_SFTP_KEYMGMT.UPDATE_PRIV_KEY.');
            DBMS_OUTPUT.put_line('was called by owner: '||v_owner||' name: '||v_name);
            DBMS_OUTPUT.put_line('Not allowing rows to be updated.');
            RETURN '1=0';
        END IF;
        --RETURN NULL;
    END user_data_update_security
    ;

    FUNCTION user_data_delete_security (owner VARCHAR2, objname VARCHAR2)
    RETURN VARCHAR2
    IS
        v_owner varchar2(1024);
        v_name varchar2(1024);
        v_lineno number;
        v_caller_t varchar2(1024);

        v_depth BINARY_INTEGER := UTL_CALL_STACK.dynamic_depth;
    BEGIN
        IF v_depth < 4 THEN
            DBMS_OUTPUT.put_line('security check found not called by LEE.AS_SFTP_KEYMGMT.DELETE_PRIV_KEY.');
            dbms_output.put_line('call stack less than 4: '||v_depth);
            DBMS_OUTPUT.put_line('Not allowing rows to be selected.');
            RETURN '1=0';
        END IF;
        /*
        FOR i in 1..v_depth
        LOOP
            v_owner := UTL_CALL_STACK.owner(i);
            v_name := UTL_CALL_STACK.concatenate_subprogram(UTL_CALL_STACK.subprogram(i));
            dbms_output.put_line('i='||i||' owner: '||v_owner||' name: '||v_name);
        END LOOP;
        */
        v_owner := UTL_CALL_STACK.owner(4);
        v_name := UTL_CALL_STACK.concatenate_subprogram(UTL_CALL_STACK.subprogram(4));
        --dbms_output.put_line('owner: '||v_owner||' name: '||v_name);
        --owa_util.who_called_me(v_owner, v_name, v_lineno, v_caller_t);
        --dbms_output.put_line('owner: '||v_owner||' name: '||v_name||' lineno: '||v_lineno||' caller_t: '||v_caller_t);
        --dbms_output.put_line(DBMS_UTILITY.format_call_stack);
        IF v_owner = 'LEE' AND v_name = 'AS_SFTP_KEYMGMT.DELETE_PRIV_KEY' THEN
            RETURN NULL;
        ELSE
            DBMS_OUTPUT.put_line('security check found not called by LEE.AS_SFTP_KEYMGMT.DELETE_PRIV_KEY.');
            DBMS_OUTPUT.put_line('was called by owner: '||v_owner||' name: '||v_name);
            DBMS_OUTPUT.put_line('Not allowing rows to be deleted.');
            RETURN '1=0';
        END IF;
        --RETURN NULL;
    END user_data_delete_security
    ;
END as_sftp_keymgmt_security;

Much of the code is repeated so I could have refactored it. Maybe later.

as_sftp_keymgmt PACKAGE BODY

Implementation of the methods to do the login and DML is anticlimatic.

CREATE OR REPLACE PACKAGE BODY as_sftp_keymgmt AS
    --
    -- This is the only method to select the private key when fine grained access control is added to the table
    -- with as_sftp_keymgt_security package. It is a package private function only called by login()
    --
    FUNCTION get_priv_key(i_host VARCHAR2, i_user VARCHAR2)
    RETURN CLOB
    IS
        v_clob CLOB;
    BEGIN
        SELECT key INTO v_clob
        FROM as_sftp_private_keys k
        WHERE host = i_host AND k.id = i_user
        ;
        RETURN v_clob;
    EXCEPTION
        WHEN NO_DATA_FOUND THEN
            raise_application_error(-20713, 'no record found in table as_sftp_private_keys for host='||i_host||', id='||i_user);
    END ;

    --
    -- The method for obtaining the private key and using it to call as_sftp.login
    --
    PROCEDURE login(i_host VARCHAR2, i_user VARCHAR2, i_passphrase VARCHAR2 := NULL, i_log_level pls_integer := null)
    IS
        v_priv_key VARCHAR2(32767) := get_priv_key(i_host, i_user);
    BEGIN
        as_sftp.login(i_log_level => i_log_level, i_user => i_user, i_priv_key => v_priv_key, i_passphrase => i_passphrase);
    END;

    --
    -- 3 DML methods for manipulating the key table records
    --
    PROCEDURE insert_priv_key(i_host VARCHAR2, i_user VARCHAR2, i_key CLOB)
    IS
    BEGIN
        INSERT INTO as_sftp_private_keys(host, id, key) VALUES(i_host, i_user, i_key);
        COMMIT;
    END insert_priv_key;

    PROCEDURE update_priv_key(i_host VARCHAR2, i_user VARCHAR2, i_key CLOB)
    IS
    BEGIN
        UPDATE as_sftp_private_keys
            SET key = i_key
            WHERE host = i_host AND id = i_user
            ;
        COMMIT;
    END update_priv_key;

    PROCEDURE delete_priv_key(i_host VARCHAR2, i_user VARCHAR2)
    IS
    BEGIN
        DELETE FROM as_sftp_private_keys
            WHERE host = i_host AND id = i_user
            ;
        COMMIT;
    END delete_priv_key;
END as_sftp_keymgmt;

Conclusion

The code shown here is available on my github page at as_sftp_keymgmt. I also forked as_ftp and put in a pull request to Mr. Scheffer. That version is more extensive and completely integrated with as_sftp. I cannot imagine supporting it going forward though if Anton does not incorporate it. You are welcome to the code though.

I am not sure I enjoyed this exercise, but I learned enough about Oracle Fine Grained Access Control to accomplish my goal. Hope it was helpful.

HTML Table Markup Redux

Tue, 18 Jan 2022 10:30:00 -0500

CSS Hell

Previously I wrote about a new package I published on github at app_html_table_pkg. My celebration was premature. The universe reared up and bit me in the hind quarters for daring to mess around with HTML again. I knew better!

It turns out that Micrsoft Outlook and Google Gmail are partially braindead with respect to CSS. They support some of it, but not all. In particular the fancy features I used for right justifying specified columns and alternating row colors were a bust.

I went back to work and retrofitted app_html_table_pkg with a p_older_css_support flag that you must use if you are sending the HTML to Outlook clients (and perhaps others).

Gmail is a complete bust. It does not even respect the older method of specifying a style class in the HTML <tr><td> tags. Bah! I’m not going to hard code everything in the HTML just to support Gmail. Most businesses that would use this are on Outlook I suspect.

With these changes, the tables display correctly (or mostly correct) in both the web and PC versions of Outlook 365 (PC version 2112 and also Version 2102). I do not know about older Outlook clients.

Google Chrome, Edge, Firefox, and Thunderbird email all support the original with p_older_css_support not set to ‘Y’, but they also handle the less elegant way just fine.

I’m not even going there on email clients on phones and tablets. Not interested. If you want to hack at it, be my guest and send a pull request if you get it working. There may be others who want it too.

HTML Table Markup in Oracle

Sun, 16 Jan 2022 10:30:00 -0500

HTML Table Markup for Oracle Query Resultset

Use Case - Generate HTML Table Markup Inside Oracle Database

Whenever a question is posted on this subject, the immediate knee-jerk reaction is to tell the hapless suppicant to use sql*plus HTML markup option. All well and good, but we do have use cases where going out to a client program from the database is not practical. One of the most common is to generate an HTML Table that is included in an email sent directly from the database.

There is a well established pattern to do this with DBMS_XMLGEN and XLSTYPE stylesheet processing. A web search will return plenty of examples.

There are a few issues with the technique, the most vexing of which is that all of the table data is left justified in the cells. If you want numbers to be right justified, the standard mechanism we reach for with TO_CHAR or LPAD(TO_CHAR to left pad the result with spaces does not work because HTML rendering ignores whitespace. We need to add a modifier to the HTML Table Data (<td>) tag to right-align the cells, but only for particular columns.

CSS Style Sheet Solution

The implemented solution can be found on my github app_html_table_pkg repository page. If you are an HTML guru, you should skip straight to the package and skip my explanation.

I have of course dabbled in HTML but most of that was in pre-historic times before CSS stylesheets gained traction. I found the CSS subject overwhelming, and it was compounded by being a moving target. Much of the information you find on the web is no longer best practice.

We do not have control of the overall document in this case, or at least not always. We may not be able to put CSS style information in the HTML Header. Luckily for us, somewhere along the way the committee to make web page development obtuse decided we could use locally scoped styles, meaning it is legal and not just a hack to put styles directly in the HTML Body. Coolio!

I complain about how the rules seem obscure and a moving target, but once I finally got a handle on enough of the pieces, it turns out to be a relatively understandable and isolated bit of HTML code that will allow us to control the formatting of our table without impacting anything else in the document.

HTML Components

div

We wrap our entire construct in <div id=”plsql-table”>…</div> to isolate it from the rest of the document.

style

Next we produce a <style type=”text/css” scoped> section encapsulating our style rules for the table.

We have directives that adjust the table border and spacing, the optional caption font and spacing, and directions for <th> table header and <td> element formatting. We can set background and foreground colors as well.

This is fairly standard HTML attribute processing that is familiar enough for most of us who are not CSS gurus.

specific column/row styles

Where the solution becomes more complex is when we direct style elements to particular columns or rows. The one to solve our specific issue of right justifying particular columns is encoded as:

    tr > td:nth-of-type(4) { text-align:right; }

Huh. That was my first thought. I’ll read that as for any table row do this for the fourth table data element (aka column). I’m not going to try to explain the rules because I’ll butcher it, but that is what it does.

A related construct that lets us alternate the background color on rows is

    tr:nth-child(odd) { background-color: AliceBlue }

That says for table row elements, on the odd rows use this background color. You could leave the even rows with the default background or make another directive for nth-child(even).

Example

Putting it all together the package can output an HTML table configured specifically for our query with any style adjustments you desire. You don’t have to understand it all. Most of the time you can copy/paste the default style and tweak it a bit. This one is on the fancy side while the default from the package is an austere black and white.

	<div id="plsql-table">
	<style type="text/css" scoped>
	
	table {
	    border: 1px solid black; 
	    border-spacing: 0; 
	    border-collapse: collapse;
	}
	caption {
	    font-style: italic;
	    font-weight: bold;
	    font-size: larger;
	    margin-bottom: 0.5em;
	}
	th {
	    text-align:left;
	    background-color: LightGrey
	}
	th, td {
	    border: 1px solid black; 
	    padding:4px 6px;
	}
	tr:nth-child(odd) { background-color: AliceBlue }
	tr > td:nth-of-type(1) {
	    text-align:right;
	}
	tr > td:nth-of-type(4) {
	    text-align:right;
	}
	</style>
	<table>
	<caption>Poorly Paid People</caption>
	<tr>
	  <th>Emp ID</th>
	  <th>Full Name</th>
	  <th>Date_x002C_Hire</th>
	  <th>Salary</th>
	</tr>
	<tr>
	  <td>999</td>
	  <td>  Baggins, Bilbo &quot;badboy&quot; </td>
	  <td>12/31/1999</td>
	  <td>     $123.45</td>
	</tr>
	<tr>
	  <td>206</td>
	  <td>Gietz, William</td>
	  <td>06/07/2002</td>
	  <td>   $8,300.00</td>
	</tr>
	<tr>
	</table></div>

Here is how it is rendered in my browser:

HTML Table Markup from PL/SQL

More CSV Fun - Turn CSV into SQL Resultset

Sun, 09 Jan 2022 10:30:00 -0500

Ad-Hoc CSV Parsing

I’ve been going down some dark holes trying to be able to use CSV data as a basis for starting an ad-hoc query. I finally arrived at something workable. While I was at it I built myself some tools for manipulating lists and strings that we can use to make PL/SQL a little less verbose for some tasks.

Consider the classic way you can hard code data in a SQL script:

WITH a AS (
    SELECT 'abc' AS col1, 123 AS col2, 'xyz' AS col3
    UNION ALL
    SELECT 'def' AS col1, 456 AS col2, 'ghi' AS col3
    UNION ALL
    SELECT 'lmn' AS col1, 789 AS col2, 'opq' AS col3
) 
SELECT * FROM a
;

Uggh, that is painful. If you have bind variables and maybe some TO_DATE functions to throw into the mix it can get even worse.

Perlish Utility Methods

There are two basic tool types provided by this User Defined Type (UDT) Object you can find at perlish_util_udt. One type are methods that work like some Perl builtin methods for manipulating lists. The other type facilitates parsing CSV data.

One of the constructor methods for perlish_util_udt takes a string as a parameter that it expects to contain Comma Separated Values (CSV) per RFC4180 (which means anything Excel puts out as CSV and anything else you are likely to find that calls itself CSV). The constructor calls perlish_util_udt.split_csv on that puppy and populates the member attribute named arr with the resulting collection of VARCHAR2 elements. Together with the perlish_util_udt.get method that takes an index parameter, this makes it possible to build the object and access those array elements in a SELECT.

Now that is all well and good for a single line of CSV, but what if we have a bunch of lines of CSV? The answer is another method in that toolbox named perlish_util_udt.split_clob_to_lines.

Together these two static methods plus that instance constructor and get method give us the tools we need to deal with that set of CSV records.

An example tells the story better than all those words:

WITH a AS (
    SELECT perlish_util_udt(t.column_value) AS p
    FROM TABLE( perlish_util_udt.split_clob_to_lines(
q'["abc",123,xyz
def,456,"ghi"
lmn,789,opq]'
                                                    )
    ) t
) -- remember you need a table alias to access object methods and attributes
-- thus making the table alias x for a here.
SELECT x.p.get(1) AS col1, x.p.get(2) AS col2, x.p.get(3) AS col3
FROM a x
;

which results in the following (where the quotes were put around the pieces by SqlDeveloper upon Text output:

"COL1"	"COL2"	"COL3"
"abc"	"123"	"xyz"
"def"	"456"	"ghi"
"lmn"	"789"	"opq"

Looking at the query in pieces

                perlish_util_udt.split_clob_to_lines(
q'["abc",123,xyz
def,456,"ghi"
lmn,789,opq]'
                                                    )

split_clob_to_line returns a collection of VARCHAR2(4000) elements. We wrap that function call in a TABLE cast (perhaps unnecessarily depending on your Oracle version), and now we can select those lines as rows from the input string.

    SELECT t.column_value AS r
    FROM TABLE( perlish_util_udt.split_clob_to_lines(
q'["abc",123,xyz
def,456,"ghi"
lmn,789,opq]'
                                                    )
    ) t
;

Showing the result in JSON may or may not help. All of the outputs from SqlDevloper want to mess with the quotes. This shows those quotes are still in the data

{
  "results" : [
    {
      "columns" : [
        {
          "name" : "R",
          "type" : "VARCHAR2"
        }
      ],
      "items" : [
        {
          "r" : "\"abc\",123,xyz"
        },
        {
          "r" : "def,456,\"ghi\""
        },
        {
          "r" : "lmn,789,opq"
        }
      ]
    }
  ]
}

Now that we have our CSV input broken into lines, we need to break the individual lines into fields. This is where we usually run into trouble with the SQL toolbox. If you start thinking about reaching for PIVOT/UNPIVOT remember that our data has natural order (we want the columns to come out in an order or with names), but there is nothing in the data itself that let’s us specify that to the SQL engine. You can use a bunch of REGEXP_SUBSTR calls to parse the pieces into fields, but c’mon man, that is worse than the problem we are trying to solve. Well, usually anyway. But if we can hide that complexity in an object that lets us split the data into fields and then access the fields by index number, we preserve our natural order without getting too gross.

That leads us to using split_csv to parse that CSV row/line into a collection:

    SELECT perlish_util_udt(t.column_value) AS p
    FROM TABLE( perlish_util_udt.split_clob_to_lines(
q'["abc",123,xyz
def,456,"ghi"
lmn,789,opq]'
                                                    )
    ) t

Showing that result as JSON we can see each row p is an object with a member attribute that is a collection:

{
  "results" : [
    {
      "columns" : [
        {
          "name" : "P",
          "type" : "LEE.PERLISH_UTIL_UDT"
        }
      ],
      "items" : [
        {
          "p" : "LEE.PERLISH_UTIL_UDT(LEE.ARR_VARCHAR2_UDT(abc, 123, xyz))"
        },
        {
          "p" : "LEE.PERLISH_UTIL_UDT(LEE.ARR_VARCHAR2_UDT(def, 456, ghi))"
        },
        {
          "p" : "LEE.PERLISH_UTIL_UDT(LEE.ARR_VARCHAR2_UDT(lmn, 789, opq))"
        }
      ]
    }
  ]
}

Then the final piece is to access individual elements of a collection. In PL/SQL you could just use the syntax

p.arr(3)

but that does not work in SQL. Fortunately, the object perlish_util_udt gives us a get function we can call with an index parameter. The only trick is that in SQL in order to access an object method or attributes, you must have a table alias for the “table” that provided the object instance in the query. In our example our “table” is named “a” and our alias for “a” is “x”.

WITH a AS (
    SELECT perlish_util_udt(t.column_value) AS p
    FROM TABLE( perlish_util_udt.split_clob_to_lines(
q'["abc",123,xyz
def,456,"ghi"
lmn,789,opq]'
                                                    )
    ) t
) -- remember you need a table alias to access object methods and attributes
-- thus making the table alias x for a here.
SELECT x.p.get(1) AS col1, x.p.get(2) AS col2, x.p.get(3) AS col3
FROM a x
;

Now, the final result this time in json:

{
  "results" : [
    {
      "columns" : [
        {
          "name" : "COL1",
          "type" : "VARCHAR2"
        },
        {
          "name" : "COL2",
          "type" : "VARCHAR2"
        },
        {
          "name" : "COL3",
          "type" : "VARCHAR2"
        }
      ],
      "items" : [
        {
          "col1" : "abc",
          "col2" : "123",
          "col3" : "xyz"
        },
        {
          "col1" : "def",
          "col2" : "456",
          "col3" : "ghi"
        },
        {
          "col1" : "lmn",
          "col2" : "789",
          "col3" : "opq"
        }
      ]
    }
  ]
}

Conclusion

Pretty handy, eh? I’ll save the other half of the functionality provided by perlish_util_udt for another day.

Perl Like Operations in PL/SQL

Sun, 02 Jan 2022 10:30:00 -0500

The Problem Needs String Hacking

I was writing a PL/SQL function to generate a MERGE sql statement from input data, the name of a table, a list of columns in the input data, and either a list of join columns or if not provided, getting them from the primary key constraint. This is something I’ve done in Perl before relatively easily, and I wanted to turn my hand to doing it in PL/SQL.

My first effort let me down the road of creating a Polymorphic Table Function to parse the CSV data. It was a nice exercise and I produced a workable PTF implementation to parse a clob and generate Oracle column data, but it is too complex for this relatively simple use case.

Next I turned towards doing it all in a set of inline PL/SQL WITH functions where I intended to have it generate the UNION ALL set of rows for the data for the MERGE USING clause. I don’t need to deploy any code in the database that my team may or may not be able to support. It winds up being a script I use to generate deployment code.

Man, what a grind. PL/SQL is very good for database things. It sucks badly for text processing like this. I found myself pining away for Perl operations like split, map and join. I’m not even going to complain about lack of variable interpolation in PL/SQL strings and how you must concatentate everything with hideous syntax because it is what it is. Yet some things we can do something about.

Consider needing to take a nested table of column names and generate the list that goes with the INSERT(…) clause of the MERGE.

    v_s := 'WHEN NOT MATCHED THEN INSERT('||v_arr_cols(1);
    FOR i IN 2..v_arr_cols.COUNT
    LOOP
        v_s := v_s||', '||v_arr_cols(i);
    END LOOP;
    v_s := v_s||')';

Phew, that is ugly. Or how about the MERGE UPDATE clause? We need to write:

WHEN MATCHED THEN UPDATE SET
    t.col1 = q.col1,
    t.col2 = q.col2

I started off writing a custom join function that also took a template string as an argument. It was sort of a meld between the Perl join and map methods.

    FUNCTION join(
        p_arr           sys.ku$_vcnt
        ,p_separator    VARCHAR2 DEFAULT ', '
        ,p_template     VARCHAR2 DEFAULT NULL
    ) RETURN VARCHAR2
    IS
        l_s     VARCHAR2(4000);
        FUNCTION apply_template(
            p_val   VARCHAR2
        )
        RETURN VARCHAR2
        IS
            l_t VARCHAR2(4000);
        BEGIN
            IF p_template IS NULL THEN
                l_t := p_val;
            ELSE 
                l_t := REPLACE(p_template, '__PLACEHOLDER__', p_val);
            END IF;
            RETURN l_t;
        END;
    BEGIN
        IF p_arr.COUNT = 0 THEN
            RETURN NULL;
        END IF;
        l_s := apply_template(p_arr(1));
        FOR i IN 2..p_arr.COUNT
        LOOP
            l_s := l_s||p_separator||apply_template(p_arr(i));
        END LOOP;
        RETURN l_s;
    END -- join
    ;

Now to write the UPDATE portion I have

    v_s := 'WHEN MATCHED THEN UPDATE SET'
        ||join(v_arr_non_pk_cols, p_separator => ',
    '
                ,p_template => 't.__PLACEHOLDER__ = q.__PLACEHOLDER__')
    ;

It did not make me happy though. It was a little too customized.

I also had some utility methods as standalone functions (split_csv and transform_perl_regexp) that really needed a package or user defined type home. I wound up creating a new User Defined Type to hold my Perlish methods. I called it japh_util_udt originally, which comes from the phrase “I’m just another perl hacker.” Most people don’t get that, so I renamed it to perlish_util_udt.

Perlish Utility User Defined Type

You can find it in my plsql_utilities github repository.

From the REAMDE.md in the repository:

perlish_util_udt

It isn’t Perl, but it makes some Perlish things a bit easier in PL/SQL. We also get handy methods for splitting Comma Separated Value (CSV) text into lines and fields, which you can use independent of the Perlish methods, and even one that turns a CSV clob into a private temporary table.

There is valid argument that when you are programming in a language you should use the facilities of that language, and that attempting to layer the techniques of another language upon it is a bad idea. I see the logic and partially agree. I expect those who later must support my work that uses this utility will curse me. Yet PL/SQL really sucks at some string and list related things. This uses valid PL/SQL object techniques to manipulate strings and lists in a way that is familiar to Perl hackers.

A perlish_util_udt object instance holds an arr_varchar2_udt collection attribute which you will use when employing the following member methods;

map
join
sort
get
combine

All member methods except get have static alternatives using arr_varchar2_udt parameters and return types, so you are not forced to use the Object Oriented syntax.

It has static method split_csv (returns arr_varchar2_udt) that formerly lived as a standalone function in the plsql_utilities library as split. We have a static method split_clob_to_lines that returns an arr_varchar2_udt collection of “records” from what is assumed to be a CSV file. It parses for CSV syntax when splitting the lines which means there can be embedded newlines in text fields in a “record”.

There is a static procedure create_ptt_csv that consumes a CLOB containing lines of CSV data and turns it into a private temporary table for your session. The PTT has column names from the first line in the CLOB.

It also has a static method named transform_perl_regexp that has nothing to do with arrays/lists, but is Perlish.

Most of the member methods are chainable which is handy when you are doing a series of operations.

Conclusion

I will work on a full demonstration to generate that deployable MERGE I described in the introduction. Seeing it in real action rather than the contrived cases of the documentation examples will show why I’m excited about it. Maybe PL/SQL can suck a little less for hacking strings and lists.

Polymorphic Table Function (PTF) for CSV (take 3)

Fri, 31 Dec 2021 10:30:00 -0500

Why Polymorphic Table Functions (PTF)?

They let you code a resultset transformation for the SQL engine without knowing at compile time either or both of

The input cursor/resultset definition.
The output cursor/resultset definition.

Ordinary Pipelined Table Functions already let you consume any type of input, but you must build your own code to figure out what is in it, generally by using DBMS_SQL. If you want to see how that is done, I have a User Defined Type Object app_dbms_sql_udt you can check out. Ordinary Pipelined Table Functions require that you define the output type at compile time.

Even if you can define your output resultset at compile time, the PTF functionality takes care of many of the details you must otherwise encode manually with DBMS_SQL. As I’ll demonstrate later, the amount of code needed to produce a CSV file generator is substantially less using a PTF than DBMS_SQL.

Polymorphic Table Function Tutorials and Examples

Oracle’s introduction to Polymorphic Table Functions is in the Database PL/SQL Language Reference 12.6 Overview of Polymorphic Table Functions. PL/SQL Packages and Types Reference has documentation for the DBMS_TF package that provides the server side implementation. When I want to really understand the types and constants I found it easier to look directly at the DBMS_TF package specification in the database using the Toad or SqlDeveloper schema browser. The types and constants are scattered through the documentation which makes them a little harder to put together than looking straight at the package spec.

The best beginner introduction to PTFs I found was by Tim Hall Polymorphic Table Functions in Oracle Database 18c. He has simple examples to lead you in gently.

There are a suite of example PTF implementations in the Oracle LiveSql collection. Enter the search term “polymorphic table function”. This article from Oracle Magazine by Chris Saxon goes along with one of them and I found it helpful.

Replication_Factor

The general design pattern for a PTF is that it transforms rows from one result set into another. By default there is a one to one relationship on the number of rows OUT to the number of rows IN. You can choose to include columns from the input row in the output row by setting the column pass_through flag to TRUE. You can add new columns to the output resultset. Whether you have any passthrough columns or not, there is still a relationship between the number of input rows and the number of output rows the function will produce from your New_Columns tables. In other words you might set all of the input columns to pass_through=FALSE, but the PTF will only produce the number of rows from your New_Columns tables that match the number of rows of your input resultset.

There is a capability to specify how many output rows there are for any given input row using DBMS_TF.row_replication procedure (or a scalar parameter). The procedure version requires an argument that is an index-by table you populate with a numeric value for every input row in that fetch_rows call. Note how carefully I stated that. You can control the number of output rows counted for each and every input row by populating an array with a numeric value for every input row. You can do that for any of the fetch_rows calls but do not have to do so for all of them. If you do not call row_replication procedure during a particular fetch_rows, then you get one to one row output.

You can have 0, 1 or more output rows for any specific input row. If a column from an input row is marked with pass_through=TRUE, you will get that value in your output if replication factor is 1. You will get it on 2 rows if the replication factor is 2, etc… If the replication factor is 0 for that input row, you will not see that value in the output. In the diagram below, the input value from row 2 will not appear in the output resultset. The number of output rows will be the sum of the replication factor values.

PTF Replication Factor

In the work I’ve done with CSV data I have been setting all input columns to pass_through=FALSE and using the replication_factor strictly as a way to tell Oracle how many output rows are in my output column arrays. When you have pass_through=TRUE you have to understand what Oracle does behind the scenes and how the two arrays match up.

Walk Through a PTF Example

We will walk through a Polymorphic Table Function and an associated package for creating a CSV file from an Oracle query. This requires Oracle 18c or better.

I have a prior effort for CSV file creation at app_csv_udt that should run under Oracle 10g or later. It is an Oracle Object Type, an approach I happen to like, but my coworkers not so much.

That earlier code is more complicated than this effort using a PTF. Oracle has taken care of most of the gnarly cursor management and bulk array processing leaving us with less to do in the PTF guts of our package. That doesn’t mean it is simple. I found the PTF functionality a struggle to grasp. The documentation is light and the examples I see published didn’t help me understand how it worked on the inside and what our design choices were as easily as I pick up many new things. When I look back at the examples different things stand out now than when I first went through them, so perhaps I just had a blind spot. I hope this article may help a few people learn about PTFs more easily.

The complete implementation can be found in the PLSQL_utilities library I maintain on github.

The PTF Components

The PTF function specification may be standalone, but it makes sense to me to include it in the package. Since it is part of the package which has a descriptive name, I named the function simply ptf. Note that there is no function body for the PTF function and no entry for it in the package body. It exists only in the package specification.
The describe function is a required element to support the PTF. It is usually not that complex (at least as far as I have seen so far).
The fetch_rows procedure is where we do most of the work to transform our input resultset to an output resultset.

Here are the PTF components of the package specification for our Example. There are a few more utility procedures and functions we will add later. The package name is app_csv_pkg.

CREATE OR REPLACE PACKAGE app_csv_pkg 
AUTHID CURRENT_USER
AS
    --
    -- All non numeric fields will be surrounded with double quotes. Any double quotes in the
    -- data will be backwacked to protect them. Newlines in the data are passed through as is
    -- which might cause issues for some CSV parsers.
    FUNCTION ptf(
        p_tab                   TABLE
        ,p_header_row           VARCHAR2 := 'Y'
        ,p_separator            VARCHAR2 := ','
        -- you can set these to NULL if you want the default TO_CHAR conversions
        ,p_date_format          VARCHAR2 := NULL
        ,p_interval_format      VARCHAR2 := NULL
    ) RETURN TABLE PIPELINED 
        TABLE -- so can ORDER the input
        --ROW 
        POLYMORPHIC USING app_csv_pkg
    ;

    -- the describe and fetch methods are used exclusively by the PTF mechanism. You cannot
    -- call them directly.
    FUNCTION describe(
        p_tab IN OUT            DBMS_TF.TABLE_T
        ,p_header_row           VARCHAR2 := 'Y'
        ,p_separator            VARCHAR2 := ','
        -- you can set these to NULL if you want the default TO_CHAR conversions
        ,p_date_format          VARCHAR2 := NULL
        ,p_interval_format      VARCHAR2 := NULL
    ) RETURN DBMS_TF.DESCRIBE_T
    ;
    PROCEDURE fetch_rows(
         p_header_row           VARCHAR2 := 'Y'
        ,p_separator            VARCHAR2 := ','
        -- you can set these to NULL if you want the default TO_CHAR conversions
        ,p_date_format          VARCHAR2 := NULL
        ,p_interval_format      VARCHAR2 := NULL
    )
    ;
...

Notice how the first argument to the function named ptf is of type TABLE. That is special in PTF land. It must be either a schema level table or view name, or else a Common Table Expression (CTE) (aka WITH clause). You cannot pass in a CURSOR or subselect. I vaguely understand the reasons for that, but not well enough to try to explain. The first argument with the same name is given to the describe function, but it has a different type.

If you have COLUMNS Pseudo-Operator arguments, they are to be the same between the PTF and describe functions (except for type – see below).

Neither the TABLE nor COLUMNS type arguments are passed to the fetch_rows procedure.

This gem from the DBMS_TF documentation says what I just said:

The arguments of the PTF function and DESCRIBE function must match, but with the type of any TABLE argument replaced with the DBMS_TF.TABLE_T descriptor type, and the type of any COLUMNS argument replaced with DBMS_TF.COLUMN_T descriptor.

All of the other arguments to all three methods are application specific and must be identical between the three methods. Even though you may not need the arguments in the describe function, the argument footprint much match what the SQL engine is going to provide in the call based on what it sees in the PTF definition.

It is interesting that our describe and fetch_rows methods are stateless. The SQL engine code that calls our methods maintains the state. When we call get/put methods that state is respected. There is an identifier called the XID that indexes that runtime state. It looks similar to the “Cursor id” value in DBMS_SQL.

The describe Function

From the package body:

    FUNCTION describe(
        p_tab IN OUT            DBMS_TF.TABLE_T
        ,p_header_row           VARCHAR2 := 'Y'
        ,p_separator            VARCHAR2 := ','
        -- you can set these to NULL if you want the default TO_CHAR conversions
        ,p_date_format          VARCHAR2 := NULL
        ,p_interval_format      VARCHAR2 := NULL
    ) RETURN DBMS_TF.DESCRIBE_T
    AS
        v_new_cols  DBMS_TF.columns_new_t;
    BEGIN
        -- stop all input columns from being in the output 
        FOR i IN 1..p_tab.column.COUNT()
        LOOP
            p_tab.column(i).pass_through := FALSE;
            p_tab.column(i).for_read := TRUE;
        END LOOP;
        -- create a single new output column for the CSV row string
        v_new_cols(1) := DBMS_TF.column_metadata_t(
                                    name    => 'CSV_ROW'
                                    ,type   => DBMS_TF.type_varchar2
                                );

        -- we will use row replication to put a header out on the first row if desired
        RETURN DBMS_TF.describe_t(new_columns => v_new_cols, row_replication => p_header_row IN ('Y','y'));
    END describe
    ;

We need to examine all of the input column values but we do not want them to be passed through to our output rows. To that end we loop through the input column list setting pass_through to FALSE and for_read to TRUE.

We will be defining a single “new” column for our output rowset. To do that we need a table of column types which we get with the type DBMS_TF.columns_new_t for our variable v_new_cols. We then populate the first and only element of that table with a column_metadata_t record with values for the column name and column type. Note that the type is a numeric we get from a constant, not the descriptive name like ‘VARCHAR2’. Digesting this bit of convoluted crafting is where it is handy to be staring at the package specification for DBMS_TF. It is easier than hunting through the type definitions in the documentation.

The thing we return from the describe function is a describe_t record. We populate new_columns with the structure we populated for our new output column named ‘CSV_ROW’. We also provide a boolean value for row_replication. We set that to TRUE if we need to produce a header row. If we do not need to produce a header row, we will have one to one correspondence of output rows from input rows so no replication is required.

The fetch_rows Procedure

fetch_rows is where most of the work is done. There is a lot to unpack here

    PROCEDURE fetch_rows(
         p_header_row           VARCHAR2 := 'Y'
        ,p_separator            VARCHAR2 := ','
        -- you can leave these NULL if you want the default TO_CHAR conversions for your session
        ,p_date_format          VARCHAR2 := NULL
        ,p_interval_format      VARCHAR2 := NULL
    ) AS
        v_env               DBMS_TF.env_t := DBMS_TF.get_env();
        v_rowset            DBMS_TF.row_set_t;  -- the input rowset of CSV rows
        v_row_cnt           BINARY_INTEGER;
        v_col_cnt           BINARY_INTEGER;
        --
        v_val_col           DBMS_TF.tab_varchar2_t;
        v_repfac            DBMS_TF.tab_naturaln_t;
        v_fetch_pass        BINARY_INTEGER := 0;
        v_out_row_i         BINARY_INTEGER := 0;

v_rowset will contain our input data for this fetch. v_val_col is a collection where we will place our output data for this fetch. v_repfac is where we MAY put a numeric value for each input row that will be 2 on the first row and 1 on all the rest. The reason we need that is to produce a header row. We need to output one more row than we have input rows, but we only need to do that on the first fetch. After that it won’t be necessary to populate or use v_repfac.

v_fetch_pass is used to determine whether or not we are on the first fetch and v_out_row_i is to keep track of the number of output rows on this fetch iteration.

        -- If the user does not want to change the NLS formats for the session
        -- but has custom coversions for this query, then we will apply them using TO_CHAR
        TYPE t_conv_fmt IS RECORD(
            t   BINARY_INTEGER  -- type
            ,f  VARCHAR2(1024)  -- to_char fmt string
        );
        TYPE t_tab_conv_fmt IS TABLE OF t_conv_fmt INDEX BY BINARY_INTEGER;
        v_conv_fmts         t_tab_conv_fmt;
        --
        FUNCTION apply_cust_conv(
            p_col_index     BINARY_INTEGER
            ,p_row_index    BINARY_INTEGER
        ) RETURN VARCHAR2
        IS
            v_s VARCHAR2(4000);
        BEGIN
            v_s := CASE WHEN v_conv_fmts.EXISTS(p_col_index) THEN
                      '"'
                        ||REPLACE(
                            CASE v_conv_fmts(p_col_index).t
                                WHEN DBMS_TF.type_date THEN 
                                    TO_CHAR(v_rowset(p_col_index).tab_date(p_row_index), v_conv_fmts(p_col_index).f)
                                WHEN DBMS_TF.type_interval_ym THEN 
                                    TO_CHAR(v_rowset(p_col_index).tab_interval_ym(p_row_index), v_conv_fmts(p_col_index).f)
                                WHEN DBMS_TF.type_interval_ds THEN 
                                    TO_CHAR(v_rowset(p_col_index).tab_interval_ds(p_row_index), v_conv_fmts(p_col_index).f)
                            END
                            , '"', '\\"'
                        ) -- backwack the dquotes if any
                        ||'"'
                    ELSE
                        DBMS_TF.col_to_char(v_rowset(p_col_index), p_row_index)
                END;
            IF SUBSTR(v_s,1,1) != '"' AND INSTR(v_s,p_separator) != 0 THEN
                v_s := '"'||v_s||'"';
            END IF;
            RETURN v_s;
        END; -- apply_cust_conv
    BEGIN

The custom conversion code is a bit ugly. The default conversion to char provided by DBMS_TF.col_to_char() is almost perfect. It takes care of putting strings into double quotes and backwacking any embedded double quotes. For Date and Interval conversions it will also place the results in double quotes; however, for those it depends on the default string coversions (which can be further muddied by NLS_DATE_FORMAT).

The other oddity can happen when NLS_NUMERIC_CHARACTERS employs a comma. If we encounter a separator character in an unquoted value, we quote it.

Other than the use of DBMS_TF.col_to_char, this is standard PL/SQL (though perhaps a bit ugly), so I’m not going to expand on what it does other than to say it converts Oracle types to strings in a user specified manner while meeting the quoting needs for CSV output.

Now with the main fetch_rows body:

        IF p_header_row IN ('Y','y') THEN
            -- We need to put out a header row, so we have to engage in replication_factor shenanigans.
            -- This is in case FETCH is called more than once. We get and put to the store
            -- the fetch count.
            -- get does not change value if not found in store so starts with our default 0 on first fetch call
            DBMS_TF.xstore_get('v_fetch_pass', v_fetch_pass); 
--dbms_output.put_line('xstore_get: '||v_fetch_pass);
        ELSE
            v_fetch_pass := 1; -- we do not need a header column. this will double as the flag
        END IF;

If we need to produce a header row, then we need to know whether this is the first fetch call or not. We use xstore_get here and xstore_put later to maintain our state between calls to fetch_rows. If we do not need a header row, set our flag variable to skip that.

        -- get the data for this fetch 
        DBMS_TF.get_row_set(v_rowset, v_row_cnt, v_col_cnt);

        -- set up for custom TO_CHAR conversions if requested for date and/or interval types
        FOR i IN 1..v_col_cnt
        LOOP
            IF (p_date_format IS NOT NULL AND v_env.get_columns(i).type = DBMS_TF.type_date)
            THEN
                v_conv_fmts(i) := t_conv_fmt(DBMS_TF.type_date, p_date_format);
            ELSIF p_interval_format IS NOT NULL 
                AND v_env.get_columns(i).type IN (DBMS_TF.type_interval_ym, DBMS_TF.type_interval_ds) 
            THEN
                v_conv_fmts(i) := t_conv_fmt(v_env.get_columns(i).type, p_interval_format);
            END IF;
        END LOOP;

We obtain the resultset data for this fetch, the number of rows and the number of columns. We then set up the custom conversion configuration if needed. Note that v_conv_fmts is sparse and possibly empty.

        IF v_fetch_pass = 0 THEN -- this is first pass and we need header row
            -- the first row of our output will get a header row plus the data row
            v_repfac(1) := 2;
            -- the rest of the rows will be 1 to 1 on the replication factor
            FOR i IN 2..v_row_cnt
            LOOP
                v_repfac(i) := 1;
            END LOOP;
            -- these names are already double quoted and Oracle will not allow a doublequote inside a column alias
            v_val_col(1) := v_env.get_columns(1).name;
            FOR j IN 2..v_col_cnt
            LOOP
                v_val_col(1) := v_val_col(1)||p_separator||v_env.get_columns(j).name; --join the column names with ,
            END LOOP;
            v_out_row_i := 1;
--dbms_output.put_line('header row: '||v_val_col(1));
        END IF;
        -- otherwise v_out_row_i is 0

On the first fetch and only when we need to produce a header row, we set up our replication factor table. As stated above we want two output rows for the first input row, then one each for all the others.

We build the header row by joining the column names with the separator character (comma most likely).

Next we loop through the input rows building the corresponding output column (we only output a single column!).

        FOR i IN 1..v_row_cnt
        LOOP
            v_out_row_i := v_out_row_i + 1;
            -- concatenate the string representations of columns with ',' separator
            -- into a single column for output on this row.
            -- col_to_char() conveniently surrounds the character representation
            -- of non-numeric fields with double quotes. If there is a double quote in
            -- that data it will backwack it. Newlines in the field are passed through unchanged.
            v_val_col(v_out_row_i) := apply_cust_conv(1, i); --DBMS_TF.col_to_char(v_rowset(1), i);
            FOR j IN 2..v_col_cnt
            LOOP
                v_val_col(v_out_row_i) := v_val_col(v_out_row_i)||p_separator||apply_cust_conv(j, i); --DBMS_TF.col_to_char(v_rowset(j), i);
            END LOOP;
        END LOOP;

If we generated a header row on this pass we submit our replication_factor table, then store our state for the next fetch pass.

        IF p_header_row IN ('Y','y') THEN    -- save for possible next fetch call
            IF v_fetch_pass = 0 THEN
                -- only on the first fetch 
                DBMS_TF.row_replication(replication_factor => v_repfac);
            END IF;
            v_fetch_pass := v_fetch_pass + 1;
            DBMS_TF.xstore_set('v_fetch_pass', v_fetch_pass);
        END IF;
        -- otherwies we did not do any replication and will get one for one with input rows

Notice that if we did not output a header row on this pass, we do not call DBMS_TF.row_replication.

And finally we tell the engine about our single output column collection.

        DBMS_TF.put_col(1, v_val_col);

    END fetch_rows;

Example Usage

An example including an ORDER BY clause for the PTF input:

WITH R AS (
    SELECT last_name||', '||first_name AS "Employee Name", hire_date AS "Hire Date", employee_id AS "Employee ID"
    FROM hr.employees
    --ORDER BY last_name, first_name
) SELECT *
FROM app_csv_pkg.ptf(R ORDER BY ("Employee Name", "Hire Date")
                        , p_date_format => 'YYYYMMDD'
                    )
WHERE rownum <= 10
;

Output (notice how the header row counts as one of the 10 rows! It is just a data record in the resultset.):

"Employee Name","Hire Date","Employee ID"
"Abel, Ellen","20040511",174
"Ande, Sundar","20080324",166
"Atkinson, Mozhe","20051030",130
"Austin, David","20050625",105
"Baer, Hermann","20020607",204
"Baida, Shelli","20051224",116
"Banda, Amit","20080421",167
"Bates, Elizabeth","20070324",172
"Bell, Sarah","20040204",192

Conclusion

The package implementation at PLSQL_utilities library adds two get_clob functions and two write_file procedures that can be passed either a CURSOR (that is expected to end with a call to SELECT * FROM app_csv_pkg.ptf(…), or a SQL string CLOB that does the same.

This implementation has almost as much functionality as my original Object Oriented version that uses DBMS_SQL, but with a LOT less code. I think this version is also easier to understand once you get over the shock and awe around learning about Polymorphic Table Functions. I hope this article reduces the impact of that flash bang.

Polymorphic Table Function for CSV (take 2)

Tue, 28 Dec 2021 10:30:00 -0500

Introduction

In a prior blog post Polymorphic Table Function to the Rescue? I used a hammer to make a Polymorphic Table Function do what I wanted. I wanted a single clob as input and multiple rows as output. I abused a PTF replication feature to get my way, but it wasn’t the right thing to do.

The general design pattern for a PTF is that it transforms rows from one result set into another, but for the most part there is a one to one relationship on the number of rows (replication feature nothwithstanding).

There is a capability to specify how many output rows there are for any given input row using DBMS_TF.row_replication procedure. The argument you provide is a table you populate with a value for every row returned by DBMS_TF.get_row_set in that fetch call. The value can be 0 meaning you do not return a value for that row. In the documentation for DBMS_TF.get_row_set is an example section titled Stack Polymorphic Table Function Example that shows a PTF named stack that does just that.

Although it is possible to do what I originally intended, it still seems klunky.

In order to conform to the more common PTF design pattern I broke my problem into two parts:

split a clob into lines
split each line into fields and output as Typed column values

It requires the user to do two steps, but it is a cleaner design that fits the pattern of other PTF functions.

The first part is achieved with an ordinary Pipelined table function that takes a CLOB as input and splits it into lines. It respects the CSV format quoting mechanism for protecting newlines in the data, so it is a little more complex than you might think.

The second part is achieved much as I did in the above mentioned blog post, but using the CSV row data as TABLE input rather than reading the CLOB directly.

From the README.md on github:

csv_to_table

Given a set of rows containing CSV strings, or a CLOB containing multiple lines of CSV strings, split the records into component column values and return a resultset that appears as if it was read from a table in your schema (or a table to which you have SELECT priv). We provide a Polymorphic Table Function for your use to achieve this.

You can either start with a set of CSV strings as rows, or with a CLOB that contains multiple lines, each of which are a CSV record. Note that this is a full blown CSV parser that should handle any records that comply with RFC4180 (See https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml).

    FUNCTION ptf(
        p_tab           TABLE
        ,p_table_name   VARCHAR2
        ,p_columns      VARCHAR2 -- csv list -- could be COLUMNS() construct instead
        ,p_date_fmt     VARCHAR2 DEFAULT NULL -- uses nls_date_format if null
        ,p_separator    VARCHAR2 DEFAULT ','
    ) RETURN TABLE
    PIPELINED ROW POLYMORPHIC USING csv_to_table_pkg
    ;

    -- public type to be returned by split_clob_to_lines PIPE ROW function
    TYPE t_csv_row_rec IS RECORD(
        s   VARCHAR2(4000)  -- the csv row
        ,rn NUMBER          -- line number in the input
    );
    TYPE t_arr_csv_row_rec IS TABLE OF t_csv_row_rec;

    --
    -- split a clob into a row for each line.
    -- Handle case where a "line" can have embedded LF chars per RFC for CSV format
    -- Throw out completely blank lines (but keep track of line number)
    --
    FUNCTION split_clob_to_lines(p_clob CLOB)
    RETURN t_arr_csv_row_rec
    PIPELINED
    ;

Example

To continue with my example from before I have a demo table from which to grab column types:

CREATE TABLE my_table_name(id number, msg VARCHAR2(1024), dt DATE);

Here is the query using the revised package:

    WITH R AS (
        SELECT *
        FROM csv_to_table_pkg.split_clob_to_lines(
q'!23, "this contains a comma (,)", 06/30/2021
47, "this contains a newline (
)", 01/01/2022

73, and we can have backwacked comma (\,),
92, what about backwacked dquote >\"<?, 12/28/2021
!'
        )
    ) SELECT *
    FROM csv_to_table_pkg.ptf(R, 'my_table_name', 'id, msg, dt', 'MM/DD/YYYY')
    ;

The blank line is ignored; however, the line numbers are maintained through the process so that errors/problems can be reported.

The resultset is as expected. Note the NULL date value on line 4. Here is a JSON representation:

{
  "results" : [
    {
      "columns" : [
        {
          "name" : "ID",
          "type" : "NUMBER"
        },
        {
          "name" : "MSG",
          "type" : "VARCHAR2"
        },
        {
          "name" : "DT",
          "type" : "DATE"
        }
      ],
      "items" : [
        {
          "id" : 23,
          "msg" : "this contains a comma (,)",
          "dt" : "06/30/2021"
        },
        {
          "id" : 47,
          "msg" : "this contains a newline (\n)",
          "dt" : "01/01/2022"
        },
        {
          "id" : 73,
          "msg" : "and we can have backwacked comma (,)",
          "dt" : ""
        },
        {
          "id" : 92,
          "msg" : "what about backwacked dquote >\\\"<?",
          "dt" : "12/28/2021"
        }
      ]
    }
  ]
}

As I mentioned this tracks the input line numbers including the blank lines that it discards. Here is an example of an error in the date conversion on the 4th row:

    WITH R AS (
        SELECT *
        FROM csv_to_table_pkg.split_clob_to_lines(
q'!23, "this contains a comma (,)", 06/30/2021
47, "this contains a newline (
)", 01/01/2022

73, and we can have backwacked comma (\,),12/24/
92, what about backwacked dquote >\"<?, 12/28/2021
!'
        )
    ) SELECT *
    FROM csv_to_table_pkg.ptf(R, 'my_table_name', 'id, msg, dt', 'MM/DD/YYYY')
    ;

The error text that it raises is:

ORA-20202: line number:4 col:3
Line: 73, and we can have backwacked comma (\,),12/24
has Oracle error: ORA-01840: input value not long enough for date format
ORA-06512: at "LEE.CSV_TO_TABLE_PKG", line 283
ORA-06512: at line 1

Getting the Code

You can find the package on my github site under repository plsql_utilities. For the moment it is in the branch named parse_csv, but I expect to merge it to main in the not too distant future.

Hope it was helpful.

Polymorphic Table Function to the Rescue?

Mon, 27 Dec 2021 10:30:00 -0500

Hard Coding Data in SQL and PL/SQL

Use Case - Populate a configuration table with multiple entries during deployment

One method is to write a set of single insert statements with values clause. Example:

INSERT INTO util_config(app_name, param_name, param_value, created_by, created_dt)
    VALUES('My App', 'email_address', 'bogus@mycompany.com', USER, SYSDATE)
;
INSERT INTO util_config(app_name, param_name, param_value, created_by, created_dt)
    VALUES('My App', 'debug_level', '0', USER, SYSDATE)
;
COMMIT;

Of course you can only run that once in any region so it makes testing your deployment difficult. It also doesn’t handle changing your mind about a value. You could do a DELETE first and that is a fine answer for our deployment task, but I prefer a MERGE.

MERGE INTO util_config c
USING (
    SELECT 'My App' AS app_name, 'email_address' AS param_name, 'bogus@mycompany.com' AS param_value FROM DUAL
    UNION ALL
    SELECT 'My App' AS app_name, 'debug_level' AS param_name, '0' AS param_value FROM DUAL
) q
ON (c.app_name = q.app_name AND c.param_name = q.param_name)
WHEN NOT MATCHED THEN INSERT(app_name, param_name, param_value, created_by, created_dt)
    VALUES(q.app_name, q.param_name, q.param_value, USER, SYSDATE)
WHEN MATCHED THEN UPDATE SET
        param_value = q.param_value
        ,updated_by = USER
        ,update_DT = sysdate
    WHERE DECODE(c.param_value, q.param_value, 0, 1) > 0 -- compares NULLs too
;
COMMIT;

All of that is quite messy though. I dislike having to write all that code just to deploy configuration table values. Sure, much of it is copy and paste after the first one, but still… In the past I have written Perl scripts that generate that code for me, but my clients are often not Perl people. It is time to scratch this itch with some PL/SQL.

I would like to have something where I can provide the table values in a CLOB, give it a set of column names and have it generate and execute the MERGE like so:

DECLARE
    v_lines CLOB := 
'My App,email_address,"bogus@mycompany.com"
My App,debug_level,0'
;
    v_column_names CLOB := 'app_name, param_name, param_value';
BEGIN
    _merge_my_config_('util_config', v_cloumn_names, v_lines);
END;

We will leave aside the created_by, created_dt, updated_by, and updated_dt columns for the moment. They are common enough that they might require special treatment. I would hope most would have triggers to populate them, or insert/update/merge procedures for populating tables that employ this pattern, but we can add logic for it later.

Parsing a CSV CLOB

There are many ways to load a CSV file into the database. You can use sqlldr or an external table. SQLcl, SqlDeveloper and Toad all provide ways to do it; however, when we are deploying code and data via a continuous improvement process, we typically can only use sqlplus scripts. That is the state of CI at the moment.

I need to parse that clob with my CSV rows and have that data available for a multi-row MERGE or INSERT statement. Let’s try using Polymorphic Table Function capability that was added in Oracle 18.

I have a single clob input and want table data output. When you read the PTF documentation and look at the examples it is clearly geared toward reading a table, not so much for writing to one. In my case I want a single CLOB as input and I want multi-row table column data as output. This is backwards from the common PTF model.

The closest I found was this Dynamic CSV to Columns Converter: Polymorphic Table Function Example by Chris Saxon. It assumes the input is a table of CSV rows though, not a single clob. That turns out to be important because the PTF model is one row of output for every row of input (or a multiple of the number of rows of input).

I started off providing an input TABLE consisting of:

WITH a AS (
    SELECT CAST('my data' AS CLOB) AS c FROM dual
) SELECT * FROM my_ptf_function(a);

but quickly ran afoul of the fact that CLOB is not a first class citizen in SQL. We can change to VARCHAR2, but then we may need more than one input row and we run into an issue figuring out the replication_factor to be able to have more output rows than input rows. It got ugly.

I finally abandonded doing anything of significance with the TABLE input parameter to the PTF. I just feed it dual as the table name and basically ignore it. All of the action happens with the other parameters to the PTF function. You may ask “well why are you using a PTF then?” The answer is that I still need that run time described field list for the output. Without that we have to define an object type or have a package type to match whatever we are doing. The PTF allows us to provide any field list and types at run time.

PTF Package Specification and Example Use

Here is the package specification. I include the PTF function inside the package even though it can be standalone.

CREATE OR REPLACE PACKAGE csv_to_table_pkg 
AUTHID CURRENT_USER
AS
    FUNCTION t(
        p_tab           TABLE
        ,p_table_name   VARCHAR2
        ,p_columns      VARCHAR2 -- csv list
        ,p_clob         CLOB
        ,p_date_fmt     VARCHAR2 DEFAULT NULL -- uses nls_date_format if null
    ) RETURN TABLE
    PIPELINED ROW POLYMORPHIC USING csv_to_table_pkg
    ;

    FUNCTION describe(
        p_tab IN OUT    DBMS_TF.TABLE_T
        ,p_table_name   VARCHAR2
        ,p_columns      VARCHAR2 -- csv list
        ,p_clob         CLOB
        ,p_date_fmt     VARCHAR2 DEFAULT NULL -- uses nls_date_format if null
    ) RETURN DBMS_TF.DESCRIBE_T
    ;
    PROCEDURE fetch_rows(
         p_table_name   VARCHAR2
        ,p_columns      VARCHAR2 -- csv list
        ,p_clob         CLOB
        ,p_date_fmt     VARCHAR2 DEFAULT NULL -- uses nls_date_format if null
    )
    ;
END csv_to_table_pkg;
/
show errors

You would call it like this:

create TABLE my_table_name(id number, msg VARCHAR2(1024), dt DATE);
--
SELECT x.*
FROM csv_to_table_pkg.t(
                      p_tab         => dual
                    , p_table_name  => 'my_table_name'
                    , p_columns     => 'id, msg, dt'
                    ,p_date_fmt     => 'MM/DD/YYYY'
                    ,p_clob         => '
23, "this contains a comma (,)", 06/30/2021
47, "this contains a newline (
)", 01/01/2022
73, and we can have backwacked comma (\,), 12/25/2021
92, what about backwacked dquote >\"<?, 12/28/2021
'
) x;
DROP TABLE my_table_name;

and the output looks as I expect. I export the resultset in JSON format from sqldeveloper to demonstrate that the data in the select list is SQL data types, not just strings. Note that the JSON formatter changed the embeded newline I threw in to the ‘\n’ representation. Cool.

{
  "results" : [
    {
      "columns" : [
        {
          "name" : "ID",
          "type" : "NUMBER"
        },
        {
          "name" : "MSG",
          "type" : "VARCHAR2"
        },
        {
          "name" : "DT",
          "type" : "DATE"
        }
      ],
      "items" : [
        {
          "id" : 23,
          "msg" : "this contains a comma (,)",
          "dt" : "06/30/2021"
        },
        {
          "id" : 47,
          "msg" : "this contains a newline (\n)",
          "dt" : "01/01/2022"
        },
        {
          "id" : 73,
          "msg" : "and we can have backwacked comma (,)",
          "dt" : "12/25/2021"
        },
        {
          "id" : 92,
          "msg" : "what about backwacked dquote >\\\"<?",
          "dt" : "12/28/2021"
        }
      ]
    }
  ]
}

p_tab

We expect the table to provide a single row (and only 1 row!) and do not care what is in it. The simplest thing to do is provide the name dual for p_tab.

p_table_name

Provide the name of a table that we expect to find in all_tab_columns. The column names should match what is in p_columns list (after it is UPPER cased). We do not do anything to the table. We merely use it to find the column data types. The expectation is that you want rows from the CSV that look like they came from or belong in that table.

p_columns

Provide a CSV separated string with a list of column names from that table. The CSV input rows must have a matching number of columns and be in the same order as that column list (which need not match the table definition order). Our output column list will be in that order by default as well.

p_clob

Provide the CSV data rows as a clob. You can use a text literal. If it exceeds 32767 characters, simply concatenate another literal until you have it all.

p_date_fmt

If your target field list contains DATE data types, the CSV values will need to be converted. If you are certain of the NLS_DATE_FORMAT in play in your session and your data matches that, you can leave it NULL. You can also alter your session to set NLS_DATE_FORMAT before calling the PTF. I would provide a date format. There are too many times I’ve stumbled upon something that changes the NLS_DATE_FORMAT of my session out from under me.

Note that this version does not support BLOB or INTERVAL data types.

PTF Package Body

The functions split and transform_perl_regexp are embedded here as package private functions. I also publish them separately as part of a library of plsql utilities that I support.

I wrote a blog post about transform_perl_regexp.

CREATE OR REPLACE PACKAGE BODY csv_to_table_pkg AS

    g_rows_regexp   VARCHAR2(32767);
    -- defined at the end of the package
    FUNCTION split (
        p_s            VARCHAR2
        ,p_separator    VARCHAR2    DEFAULT ','
        ,p_keep_nulls   VARCHAR2    DEFAULT 'N'
        ,p_strip_dquote VARCHAR2    DEFAULT 'Y' -- also unquotes \" and "" pairs within the field to just "
    ) RETURN arr_varchar2_udt DETERMINISTIC
    ;

    FUNCTION describe(
        p_tab IN OUT    DBMS_TF.TABLE_T
        ,p_table_name   VARCHAR2
        ,p_columns      VARCHAR2 -- csv list
        ,p_clob         CLOB
        ,p_date_fmt     VARCHAR2 DEFAULT NULL -- uses nls_date_format if null
    ) RETURN DBMS_TF.DESCRIBE_T
    AS
        v_new_cols  DBMS_TF.columns_new_t;
        v_col_names arr_varchar2_udt;

        TYPE t_col_order IS TABLE OF BINARY_INTEGER INDEX BY VARCHAR2(128);
        v_col_order t_col_order;
    BEGIN
        IF p_tab.column.COUNT() != 1 
        THEN
            RAISE_APPLICATION_ERROR(-20000,'Input table to csv_to_table_pkg.t should be table DUAL');
        END IF;
        p_tab.column(1).pass_through := FALSE;
        p_tab.column(1).for_read := TRUE;

        v_col_names := split(UPPER(p_columns));
        -- we need a hash to get from the column name to the index for both input csv order and output field order
        FOR i IN 1..v_col_names.COUNT
        LOOP
            v_col_order(v_col_names(i)) := i;
        END LOOP;

        FOR r IN (
            SELECT c.column_value AS column_name
                ,CASE WHEN a.data_type LIKE 'TIMESTAMP%' THEN 'TIMESTAMP' ELSE a.data_type END AS data_type
            FROM TABLE(v_col_names) c
            LEFT OUTER JOIN all_tab_columns a
                ON a.table_name = UPPER(p_table_name)
                    AND a.column_name = c.column_value
        ) LOOP
            IF r.data_type IS NULL THEN
                RAISE_APPLICATION_ERROR(-20001,'table: '||p_table_name||' does not have a column named '||r.column_name);
            END IF;
            IF r.data_type = 'BLOB' OR r.data_type LIKE 'INTERVAL%' THEN
                RAISE_APPLICATION_ERROR(-20002,'table: '||p_table_name||' column '||r.column_name||' is unsupported data type '||r.data_type);
            END IF;
            -- we create these in any order, but they must be in the right location in the array
            v_new_cols(v_col_order(r.column_name)) := DBMS_TF.column_metadata_t(
                                        name    => r.column_name
                                        ,type   => CASE r.data_type
                                                    WHEN 'TIMESTAMP'        THEN DBMS_TF.type_timestamp
                                                    WHEN 'BINARY_DOUBLE'    THEN DBMS_TF.type_binary_double
                                                    WHEN 'BINARY_FLOAT'     THEN DBMS_TF.type_binary_float
                                                    --WHEN 'BLOB'             THEN DBMS_TF.type_blob
                                                    WHEN 'CHAR'             THEN DBMS_TF.type_char
                                                    WHEN 'CLOB'             THEN DBMS_TF.type_clob
                                                    WHEN 'DATE'             THEN DBMS_TF.type_date
                                                    WHEN 'NUMBER'           THEN DBMS_TF.type_number
                                                    ELSE                         DBMS_TF.type_varchar2
                                                   END
                                    );
        END LOOP;

        -- we have 1 input row and MANY output rows, so replication is true
        RETURN DBMS_TF.describe_t(new_columns => v_new_cols, row_replication => TRUE);
    END describe
    ;

    PROCEDURE fetch_rows(
         p_table_name   VARCHAR2
        ,p_columns      VARCHAR2 -- csv list
        ,p_clob         CLOB
        ,p_date_fmt     VARCHAR2 DEFAULT NULL -- uses nls_date_format if null
    ) AS
        v_env               DBMS_TF.env_t := DBMS_TF.get_env(); -- put_columns.count
        v_rowset            DBMS_TF.row_set_t;
        v_in_row_count      BINARY_INTEGER;
        --
        v_rowset_out        DBMS_TF.row_set_t;
        v_col_out_cnt       BINARY_INTEGER;
        v_output_col_type   BINARY_INTEGER;
        --
        v_row               CLOB;
        v_row_cnt           BINARY_INTEGER;
        v_col_strings       arr_varchar2_udt;

        -- input row numbers including blank lines that can be skipped
        g_row_num           NUMBER := 0;
        v_rows_out          BINARY_INTEGER := 0;
    BEGIN
        -- the number of columns in our output rows should match number of csv fields
        v_col_out_cnt := v_env.put_columns.COUNT();

        -- in case FETCH is called more than once (unlikely)
        -- get does not change value if not found in store so starts with our default 0
        --DBMS_TF.xstore_get('g_row_num', g_row_num); 

        DBMS_TF.get_row_set(v_rowset, row_count => v_in_row_count);
        IF v_in_row_count != 1 THEN
            RAISE_APPLICATION_ERROR(-20007,'input table should only have 1 placeholder row. Use DUAL');
        END IF;
        -- we need to use count because our regexp will match an empty row. We will skip
        -- the empty row but we need to line number to help with debug error message
        v_row_cnt := REGEXP_COUNT(p_clob, g_rows_regexp) - 1; -- one extra matches on $
--dbms_output.put_line('got '||v_row_cnt||' rows from clob');
        -- loop over the rows split from the input string
        FOR i IN 1..v_row_cnt
        LOOP
            g_row_num := g_row_num + 1;
            -- pull a line out of the text input (sans newline)
            v_row := REGEXP_SUBSTR(p_clob, g_rows_regexp, 1, i, NULL, 1);
--dbms_output.put_line('row '||g_row_num||' : '||v_row);
            -- split the row into csv fields stripping dquotes and unquoting chars inside dquotes
            v_col_strings := split(v_row);
            IF v_col_strings.COUNT = 0 THEN
                -- just skip empty rows now that we captured the rownumber
--dbms_output.put_line('row '||g_row_num||' had 0 csv columns');
                CONTINUE;
            ELSIF v_col_strings.COUNT != v_col_out_cnt THEN
                RAISE_APPLICATION_ERROR(-20003,'row '||g_row_num||' has cnt='||v_col_strings.COUNT||' csv fields, but we need '||v_col_out_cnt||' columns
ROW: '||v_row);
            END IF;
--dbms_output.put_line('row '||g_row_num||' had '||v_col_strings.COUNT||' csv columns');
            -- populate the output rowset column tables for this row
            v_rows_out := v_rows_out + 1;
            FOR j IN 1..v_col_out_cnt
            LOOP
                v_output_col_type := v_env.put_columns(j).TYPE;
                IF v_output_col_type = DBMS_TF.type_timestamp THEN
                    -- better set nls value yourself because we just shoving the string in with default conversion
                    -- likely not to ever be used
                    v_rowset_out(j).tab_timestamp(v_rows_out) := v_col_strings(j);
                ELSIF v_output_col_type = DBMS_TF.type_binary_double THEN
                    v_rowset_out(j).tab_binary_double(v_rows_out) := v_col_strings(j);
				ELSIF v_output_col_type = DBMS_TF.type_binary_float THEN
                    v_rowset_out(j).tab_binary_float(v_rows_out) := v_col_strings(j);
				ELSIF v_output_col_type = DBMS_TF.type_char THEN
                    v_rowset_out(j).tab_char(v_rows_out) := v_col_strings(j);
				ELSIF v_output_col_type = DBMS_TF.type_clob THEN
                    v_rowset_out(j).tab_clob(v_rows_out) := v_col_strings(j);
				ELSIF v_output_col_type = DBMS_TF.type_date THEN
                    IF p_date_fmt IS NULL THEN
                        v_rowset_out(j).tab_date(v_rows_out) := v_col_strings(j); -- default to nls_date_fmt
                    ELSE
                        v_rowset_out(j).tab_date(v_rows_out) := TO_DATE(v_col_strings(j), p_date_fmt);
                    END IF;
				ELSIF v_output_col_type = DBMS_TF.type_number THEN
                    v_rowset_out(j).tab_number(v_rows_out) := v_col_strings(j);
                ELSE -- in describe we made sure the only thing left is varchar2
                    v_rowset_out(j).tab_varchar2(v_rows_out) := v_col_strings(j);
                END IF;
            END LOOP; -- end loop on columns
        END LOOP; -- end loop on newline separated rows in clob

        --DBMS_TF.xstore_set('g_row_num', g_row_num);
        -- we got a single row of input, but are now writing v_rows_out records output.
        -- The only way to do that is with the funky replication_factor. It was not designed
        -- for this, but it works.
        DBMS_TF.put_row_set(v_rowset_out, replication_factor => v_rows_out);
    END fetch_rows;

    FUNCTION transform_perl_regexp(p_re VARCHAR2)
    RETURN VARCHAR2
    DETERMINISTIC
    IS
        /*
            strip comment blocks that start with at least one blank, then
            '--' or '#', then everything to end of line or string
        */
        c_strip_comments_regexp CONSTANT VARCHAR2(32767) := '[[:blank:]](--|#).*($|
    )';
    BEGIN
      -- note that \n and \t will be replaced if not preceded by a \
      -- \\n and \\t will not be replaced. Unfortunately, neither will \\\n or \\\t.
      -- We are not parsing into tokens, so this is as close as we can get cheaply
      RETURN 
        REGEXP_REPLACE(
          REGEXP_REPLACE(
            REGEXP_REPLACE(
              REGEXP_REPLACE(
                REGEXP_REPLACE(p_re, c_strip_comments_regexp, NULL, 1, 0, 'm') -- strip comments
                    , '\s+', NULL, 1, 0                 -- strip spaces and newlines too like 'x' modifier
              ) 
              , '(^|[^\\])\\t', '\1'||CHR(9), 1, 0    -- replace \t with tab character value so it works like in perl
            ) 
            , '(^|[^\\])\\n', '\1'||CHR(10), 1, 0       -- replace \n with newline character value so it works like in perl
          )
          , '(^|[^\\])\\r', '\1'||CHR(13), 1, 0         -- replace \r with CR character value so it works like in perl
        ) 
      ;
    END transform_perl_regexp;


  FUNCTION split (
     p_s            VARCHAR2
    ,p_separator    VARCHAR2    DEFAULT ','
    ,p_keep_nulls   VARCHAR2    DEFAULT 'N'
    ,p_strip_dquote VARCHAR2    DEFAULT 'Y' -- also unquotes \" and "" pairs within the field to just "
  ) RETURN arr_varchar2_udt DETERMINISTIC
-- when p_s IS NULL, returns initialized collection with COUNT=0
--
/*

Treat input string p_s as following the Comma Separated Values (csv) format 
(not delimited, but separated) and break it into an array of strings (fields) 
returned to the caller. This is overkill for the most common case
of simple separated strings that do not contain the separator char and are 
not quoted, but if they are double quoted fields, this will handle them 
appropriately including the quoting of " within the field.

We comply with RFC4180 on CSV format (for what it is worth) while also 
handling the mentioned common variants like backwacked quotes and 
backwacked separators in non-double quoted fields that Excel produces.

See https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml

*/
--
/*
MIT License

Copyright (c) 2021 Lee Lindley

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
  IS
        v_str       VARCHAR2(32767);    -- individual parsed values cannot exceed 4000 chars
        v_occurence BINARY_INTEGER := 1;
        v_i         BINARY_INTEGER := 0;
        v_cnt       BINARY_INTEGER;
        v_arr       arr_varchar2_udt := arr_varchar2_udt();

        -- we are going to match multiple times. After each match the position 
        -- will be after the last separator.
        v_regexp    VARCHAR2(128) := transform_perl_regexp('
\s*                         -- optional whitespace before anything, or after
                            -- last delim
                            --
   (                        -- begin capture of \1 which is what we will return.
                            -- It can be NULL!
                            --
       "                        -- one double quote char binding start of the match
           (                        -- just grouping
 --
 -- order of these next things matters. Look for longest one first
 --
               ""                       -- literal "" which is a quoted quote 
                                        -- within dquote string
               |                        
               \\"                      -- Then how about a backwacked double
                                        -- quote???
               | 
               [^"]                     -- char that is not a closing quote
           )*                       -- 0 or more of those chars greedy for
                                    -- field between quotes
                                    --
       "                        -- now the closing dquote 
       |                        -- if not a double quoted string, try plain one
 --
 -- if the capture is not going to be null or a "*" string, then must start 
 -- with a char that is not a separator or a "
 --
       [^"'||p_separator||']    -- so one non-sep, non-" character to bind 
                                -- start of match
                                --
           (                        -- just grouping
 --
 -- order of these things matters. Look for longest one first
 --
               \\'||p_separator||'      -- look for a backwacked separator
               |                        
               [^'||p_separator||']     -- a char that is not a separator
           )*                       -- 0 or more of these non-sep, non backwack
                                    -- sep chars after one starting (bound) a 
                                    -- char 1 that is neither sep nor "
                                    --
   )?                       -- end capture of our field \1, and we want 0 or 1
                            -- of them because we can have ,,
 --
 -- Since we allowed zero matches in the above, regexp_subst can return null
 -- or just spaces in the referenced grouped string \1
 --
   ('||p_separator||'|$)    -- now we must find a literal separator or be at 
                            -- end of string. This separator is included and 
                            -- consumed at the end of our match, but we do not
                            -- include it in what we return
');
--
--
--
--v_log app_log_udt := app_log_udt('TEST');
  BEGIN
        IF p_s IS NULL THEN
            RETURN v_arr; -- will be empty
        END IF;
--v_log.log_p(TO_CHAR(REGEXP_COUNT(p_s,v_regexp)));
        -- since our matched group may be a null string that regexp_substr 
        -- returns before we are done, we cannot rely on the condition that 
        -- regexp_substr returns null to know we are done. 
        -- That is the traditional way to loop using regexp_subst, but that 
        -- will not work for us. So, we first have to find out how many fields
        -- we have including null captures and run regexp_substr that many times
        v_cnt := REGEXP_COUNT(p_s, v_regexp);
        --
        -- A "delimited" string, as opposed to a separated string, will end in
        -- a delimiter char. In other words there is always one "delimiter"
        -- after every field. But the most common case of CSV is a "separator"
        -- style which does not have separator at the end, and if we actually
        -- have a separator at the end of the string, it is because the last
        -- field value was NULL!!!! In that scenario with the trailing separator
        -- we want to count that NULL and include it in our array.
        -- In the case where the last char is not a "separator" char, 
        -- the regexp will match one last time on the zero-width $. That is an
        -- oddity of how it is constructed.
        -- For our purposes of expecting a separated string, not delimited,
        -- we need to reduce the count by 1 for the case where the last
        -- character is NOT a separator. 
        --
        -- I do not know what to say. I was very dissapointed I could not handle
        -- all the logic in the regexp, but Oracle regexp are just not as
        -- powerful as the ones in Perl which have negative/positive lookahead
        -- and lookbehinds plus the concept of "POS()" so you can match against
        -- the start of the substr like ^ does for the whole thing. Without
        -- those features, this was very difficult. It is also possible I am
        -- missing something important and a regexp expert could do it more
        -- simply. I would not mind being corrected and shown a better way.
        --
        IF SUBSTR(p_s,-1,1) != p_separator THEN -- if last char of string is not the separator
            v_cnt := v_cnt - 1;
        END IF;

        FOR v_occurence IN 1..v_cnt
        LOOP
            v_str := REGEXP_SUBSTR(
                    p_s                 -- the string we are parsing
                    ,v_regexp           -- the regexp we built using the chosen separator (like ',')
                    ,1                  -- starting at the beginning of the string on the first call
                    ,v_occurence        -- but on subsequent calls we will get 2nd, then 3rd, etc, match of the pattern
                    ,''                 -- no regexp modifiers
                    ,1                  -- we want the \1 grouping match returned, not the entire expression
            );
--v_log.log_p(TO_CHAR(v_occurence)||' x'||v_str||'x');

            -- cannot use this test for NULL like the man page for regexp_substr
            -- shows because our grouped match can be null as a valid value
            -- while still parsing the string.
            --EXIT WHEN v_str IS NULL;

            v_str := TRIM(v_str);                               -- if it is a double quoted string, can still have leading/trailing spaces in the value
            IF v_str IS NOT NULL OR p_keep_nulls = 'Y' THEN     -- otherwise it was an empty string which we discard.
                -- we WILL add this to the array
                IF v_str IS NULL THEN
                    NULL;
                ELSIF SUBSTR(v_str,1,1) = '"' THEN                 -- it IS a double quoted string
                    IF p_strip_dquote = 'Y' THEN
                        -- get rid of starting and ending " char
                        -- replace any \" or "" pairs with single "
                        v_str := REGEXP_REPLACE(v_str, 
                                    '^"|"$'         -- leading " or ending "
                                    ||'|["\\]'  -- or one of chars " or \
                                        ||'(")'     -- that is followed by a " and we capture that one in \1
                                    ,'\1'           -- We put any '"' we captured back without the backwack or " quote
                                    ,1              -- start at position 1 in v_str
                                    ,0              -- 0 occurence means replace all of these we find
                                ); 
                    END IF;
                ELSE 
                        -- not a double quoted string so unbackwack separators inside it. Excel format
                        v_str := REGEXP_REPLACE(v_str, '\\('||p_separator||')', '\1', 1, 0);
                END IF; -- end if double quoted string
                -- Note that if it was an empty double quoted string we are still putting it into the array.
                -- So, you can still get nulls in the case they are given to you as "" and we stripped the dquotes,
                -- even if you asked to not keep nulls. Cause an empty string is not NULL. Uggh.
                v_i := v_i + 1;
                v_arr.EXTEND;
                -- this will raise an error if the value is more than 400 chars
                v_arr(v_i) := v_str;
            END IF; -- end not an empty string or we want to include NULL
        END LOOP;
        RETURN v_arr;
  END split
    ;
-- start package initialization block
BEGIN

    g_rows_regexp   := transform_perl_regexp('
(                               # capture in \1
  (                             # going to group 0 or more of these things
    [^"\n\\]+                   # any number of chars that are not dquote, backwack or newline
    |
    (                           # just grouping for repeat
        \\ \n                   # or a backwacked \n but put space between them so gets transformed correctly
    )+                          # one or more protected newlines (as if they were in dquoted string)
    |
    (                           # just grouping for repeat
        \\"                     # or a backwacked "
    )+                          # one or more protected "
    |
    "                           # double quoted string start
        (                       # just grouping. Order of the next set of things matters. Longest first
            ""                  # literal "" which is a quoted dquoute within dquote string
            |
            \\"                 # a backwacked dquote 
            |
            [^"]                # any single character not the above two multi-char constructs, or a dquote
                                #     Important! This can be embedded newlines too!
        )*                      # zero or more of those chars or constructs 
    "                           # closing dquote
    |                           
    \\                          # or a backwack, but do this last as it is the smallest and we do not want
                                #   to consume the backwack before a newline or a dquote
  )*                            # zero or more strings on a single "line" that could include newline in dquotes
                                # or even a backwacked newline
)                               # end capture \1
(                               # just grouping 
    $|\n                        # require match newline or string end 
)                               # close grouping
');

END csv_to_table_pkg;
/
show errors

Using the PTF for Data Deploy

Here is how I could deploy the rows in my example at the start of this blog post:

    MERGE INTO util_config
    USING (
        SELECT *
        FROM csv_to_table_pkg.t(
            p_tab       => dual
            ,p_table_name   => 'util_config'
            ,p_columns      => 'app_name, param_name, param_value'
            ,p_clob         => '
My App, email_address, bogus@mycompany.com
My App, debug_level, 0
'
        )
    ) q
    ON (c.app_name = q.app_name AND c.param_name = q.param_name)
WHEN NOT MATCHED THEN INSERT(app_name, param_name, param_value, created_by, created_dt)
    VALUES(q.app_name, q.param_name, q.param_value, USER, SYSDATE)
WHEN MATCHED THEN UPDATE SET
        param_value = q.param_value
        ,updated_by = USER
        ,update_DT = sysdate
    WHERE DECODE(c.param_value, q.param_value, 0, 1) > 0 -- compares NULLs too
;
COMMIT;

Our use case was to simplify deploy of DATA to configuration tables. We certainly seemed to have done a crapton of hard work to make this easier. I do not know whether the solution is worth it or not.

I will likely add this package to plsql utilities, but have not done so yet. I need to exercise it a bit more to see if anything shakes out. I’m also still uncomfortable with this pattern of abusing the replication factor for PTF. It does not really fit the PTF design. I might break this into two pieces, one of which parses the clob into lines and the other which uses a PTF to parse the CSV lines as was done by Mr. Saxon in his example.

If you are exploring Polymorphic Table Functions, I hope you might find this exercise helpful. We certainly need more examples of them to help learn about it. This was not a gimme.

Perl REgexp vs Oracle

Sun, 26 Dec 2021 10:30:00 -0500

Writing Regular Expressions for Oracle As If You Were in a Perl Script

I am accustomed to writing regular expressions for Perl. It has gotten to the point where I’m super annoyed when I write regular expressions for Oracle because of both how strings are handled and lack of comment support in the RE itself. I prefer building the regular expression in small pieces with assembler style comments on the right side to remind myself what it does. I also favor using ‘\n’ and ‘\t’ to represent newline and tab rather than having to hardcode these values in the regexp string. Having a newline in the middle of an Oracle regular expression is particularly disconcerting.

Likewise building the RE by concatentating quoted string pieces and having regular SQL comments out on the right is less than satisfying. It gets ugly with all of the “||’ whatever’”, especially with embedded newlines or even concatenating a constant that contains a newline. These may seem trivial concerns, but I’m not changing.

The following Function transforms a regular expression in a manner similar to the way Perl would treat the ‘x’ modifier stripping comments and whitespace, plus allowing us to use ‘\n’, ‘\r’ and ‘\t’ as character representations that are not stripped as whitespace.

    FUNCTION transform_perl_regexp(p_re VARCHAR2)
    RETURN VARCHAR2
    IS
        /*
            strip comment blocks that start with at least one blank, then
            '--' or '#', then everything to end of line or string
        */
        c_strip_comments_regexp CONSTANT VARCHAR2(32767) := '[[:blank:]](--|#).*($|
)';
    BEGIN
    -- note that \n, \r and \t will be replaced if not preceded by a \
    -- \\n and \\t will not be replaced. Unfortunately, neither will \\\n or \\\t.
    -- If you need \\\n, use \\ \n since the space will be removed.
    -- We are not parsing into tokens, so this is as close as we can get cheaply
      RETURN 
        REGEXP_REPLACE(
          REGEXP_REPLACE(
            REGEXP_REPLACE(
              REGEXP_REPLACE(
                REGEXP_REPLACE(p_re, c_strip_comments_regexp, NULL, 1, 0, 'm') -- strip comments
                , '\s+', NULL, 1, 0                 -- strip spaces and newlines too like 'x' modifier
              ) 
              , '(^|[^\\])\\t', '\1'||CHR(9), 1, 0    -- replace \t with tab character value so it works like in perl
            ) 
            , '(^|[^\\])\\n', '\1'||CHR(10), 1, 0       -- replace \n with newline character value so it works like in perl
          )
          , '(^|[^\\])\\r', '\1'||CHR(13), 1, 0         -- replace \r with CR character value so it works like in perl
        ) 

      ;

    END;

The function transform_perl_regexp is distributed in a utility libray I support. You can use it without attribution.

Below is an example of writing a somewhat convoluted regular expression to parse a multi-line CLOB that contains CSV “records” that may contain embedded newlines with quoting as described in RFC4180.

I have more about that in the plsql_utilities library if it interests you, but the main point is to show the regular expression presentation in the Perl style and how it is transformed.

DECLARE
    v_lines     CLOB := 
------------------------
'23, "this contains a comma (,)", 06/30/2021
47, "this contains a newline (
)", 01/01/2022

73, and we can have backwacked comma (\,), 12/25/2021
92, what about backwacked dquote >\"<?, 12/28/2021'
;
------------------------

    -- beware this is using Perl-like regexp so we have to apply a transform
    v_regexp    VARCHAR2(32767) := '
(                               # capture in \1
  (                             # going to group 0 or more of these things
    [^"\n\\]+                   # any number of chars that are not dquote, backwack or newline
    |
    (                           # just grouping for repeat
        \\ \n                   # or a backwacked \n but put space between them so gets transformed correctly
    )+                          # one or more protected newlines (as if they were in dquoted string)
    |
    (                           # just grouping for repeat
        \\"                     # or a backwacked "
    )+                          # one or more protected "
    |
    "                           # double quoted string start
        (                       # just grouping. Order of the next set of things matters. Longest first
            ""                  # literal "" which is a quoted dquoute within dquote string
            |
            \\"                 # a backwacked dquote 
            |
            [^"]                # any single character not the above two multi-char constructs, or a dquote
                                #     Important! This can be embedded newlines too!
        )*                      # zero or more of those chars or constructs 
    "                           # closing dquote
    |                           
    \\                          # or a backwack, but do this last as it is the smallest and we do not want
                                #   to consume the backwack before a newline or a dquote
  )*                            # zero or more strings on a single "line" that could include newline in dquotes
                                # or even a backwacked newline
)                               # end capture \1
(                               # just grouping 
    $|\n                        # require match newline or string end 
)                               # close grouping
';
    v_line      VARCHAR2(32767);
    v_i         BINARY_INTEGER := 1;
    v_cnt       BINARY_INTEGER;
BEGIN
    v_regexp := transform_perl_regexp(v_regexp);
    DBMS_OUTPUT.put_line('preprocessed regexp: >'||v_regexp||'<');
    --
    v_cnt := REGEXP_COUNT(v_lines, v_regexp) - 1;   -- get number of times regexp will match less match on $
    FOR v_i IN 1 ..v_cnt                            -- for each time we know it will match including NULL match values
    LOOP
        v_line := REGEXP_SUBSTR(v_lines, v_regexp, 1, v_i, NULL, 1);
        dbms_output.put_line('line '||TO_CHAR(v_i)||' is >'||v_line||'<');
    END LOOP;
END;
/

Output:

preprocessed regexp: >(([^"
\\]+|(\\\n)+|(\\")+|"(""|\\"|[^"])*"|\\)*)($|
)<
line 1 is >23, "this contains a comma (,)", 06/30/2021<
line 2 is >47, "this contains a newline (
)", 01/01/2022<
line 3 is ><
line 4 is >73, and we can have backwacked comma (\,), 12/25/2021<
line 5 is >92, what about backwacked dquote >\"<?, 12/28/2021<

transform_perl_regexp is NOT a sophisticated parser. If you want to have the strings ‘ –’ (space dash dash) or ‘ #’ (space hash) appear in your regular expression, you will have to get creative or not use it. (HINT: you can put one of the characters in character class brackets). Likewise when looking for the ‘\n’ and friends, we are not actually parsing anything, so it gets messy. I’ve protected ‘\\n’, but not dealt with ‘\\\n’. These are easier though because you can separate ‘\’ from ‘\n’ with a space that is going to be stripped. It works the way I expect most of the time.

Conclusion

Regular expressions, even those you wrote last week, take signficant effort for your brain to parse. It is worthwhile to have them well documented in the code. A way to do that in Perl has been established and is used by many serious practitioners. I much prefer it and can now use it in Oracle SQL and PL/SQL programs.

Extracting BLOB from Oracle with Sqlplus

Sat, 18 Dec 2021 10:30:00 -0500

How do we get a BLOB from the database to a client file?

I wrote about using a WITH clause function for various purposes Why do I need Inline PL/SQL Methods?, one example of which was extracting a BLOB that contained an XLSX file. In that article I blithely waved my hands and said use Toad or SqlDeveloper to then extract the output into a file.

What do we do if we want to script this? There are options. You can send email from inside the database and attach the blob (see html_email), but if your goal is to automatically deliver it to a shared drive or other file destination, email is not a good option.

You can use pretty much any language that supports a robust Oracle database interface to do it from a client ETL server. Pro-C, Java, Perl DBI::DBD::Oracle, and I presume Python all have the capability to extract a BLOB from the database and write to the local filesystem. I also assume the big commercial ETL packages like Ab-initio and Informatica will do it, but I haven’t looked.

Maybe you do not have access to or expertise in these tools. The Oracle client library has SQLcl, which can be scripted to download a BLOB as mentioned in this article Using SQLcl to write out BLOBs to files in 20 lines of js. But it turns out that you don’t always have sqlcl or even Java installed on an ETL server. I know that seems odd, but I’ve seen it enough I’m no longer surprised.

I wanted to do it with sqlplus because it is almost universally available at every client and on every ETL server. Unfortunately sqlplus does not support BLOB data type as a local variable for binding. Drat. When I searched I hit the article Download BLOBs with SQLPlus by Ottmar Gobrect. He spreads plenty of credit around for the idea. The crux of the solution is to convert from BLOB, which sqlplus does not support, to CLOB which it does. How, you ask? The same way we do it when we attach a binary file to an email. We base64 encode it. Clever!

Extracting a BLOB with Sqlplus

Reusing my spreadsheet example from the article mentioned at the top, and a base64encode function published by Tim Hall we have a sqlplus script that generates and extracts a BLOB to a file as base64 encoded text.

-- 16mb should be plenty for most spreadsheets.
set long 16777216
set longchunksize 32767
set heading off
set verify off 
set feedback off 
set trimout on 
set trimspool on 
set pagesize 0 
set linesize 1000 
whenever sqlerror exit failure
-- sqlplus supports clob variables but not blob
variable vo_clob clob
--
DECLARE
    c SYS_REFCURSOR;

    FUNCTION base64encode(p_blob IN BLOB)
    RETURN CLOB
    -- -----------------------------------------------------------------------------------
    -- File Name    : https://oracle-base.com/dba/miscellaneous/base64encode.sql
    -- Author       : Tim Hall
    -- Description  : Encodes a BLOB into a Base64 CLOB.
    -- Last Modified: 09/11/2011
    -- -----------------------------------------------------------------------------------
    IS
        l_clob CLOB;
        l_step PLS_INTEGER := 12000; -- make sure you set a multiple of 3 not higher than 24573
    BEGIN
        FOR i IN 0 .. TRUNC((DBMS_LOB.getlength(p_blob) - 1 )/l_step) LOOP
            l_clob := l_clob || UTL_RAW.cast_to_varchar2(UTL_ENCODE.base64_encode(DBMS_LOB.substr(p_blob, l_step, i * l_step + 1)));
        END LOOP;
        RETURN l_clob;
    END;

    FUNCTION get_xlsx(p_src SYS_REFCURSOR) 
    RETURN CLOB AS
        v_blob          BLOB;
        v_ctxId         ExcelGen.ctxHandle;
        v_sheetHandle   BINARY_INTEGER;
    BEGIN
        v_ctxId := ExcelGen.createContext();
        v_sheetHandle := ExcelGen.addSheetFromCursor(v_ctxId, 'Employee Salaries', p_src, p_sheetIndex => 1);
        -- freeze the top row with the column headers
        ExcelGen.setHeader(v_ctxId, v_sheetHandle, p_frozen => TRUE);
        -- style with alternating colors on each row. 
        ExcelGen.setTableFormat(v_ctxId, v_sheetHandle, 'TableStyleLight11');
        -- single column format on the salary column. The ID column keeps default format
        ExcelGen.setColumnFormat(
            p_ctxId     => v_ctxId
            ,p_sheetId  => v_sheetHandle
            ,p_columnId => 5        -- the salary column
            ,p_format   => '$#,##0.00'
        );
        v_blob := ExcelGen.getFileContent(v_ctxId);
        ExcelGen.closeContext(v_ctxId);
        RETURN base64encode(v_blob);
    END;
BEGIN

    OPEN c FOR WITH add_bilbo AS (
            SELECT e.employee_id AS employee_id, e.last_name, e.first_name, d.department_name, e.salary
            FROM hr.employees e
            INNER JOIN hr.departments d
                ON d.department_id = e.department_id
            UNION ALL
            SELECT 999 AS employee_id, 'Baggins' As last_name, 'Bilbo' as first_name, 'Sales' AS department_name
                ,123.45 AS salary
            FROM dual
        ) SELECT * FROM add_bilbo ORDER BY last_name, first_name
    ;
    -- assign the uuencoded clob data to the sqlplus bind variable
    :vo_clob := get_xlsx(c);
END;
/
-- spool out the encoded clob
set termout off
spool "x.xlsx.base64"
print vo_clob
spool off
set termout on
--  base64 -d -i x.xlsx.base64 >x.xlsx && rm x.xlsx.base64
-- OR on windows
-- certutil -decode x.xlsx.base64 x.xlsx
-- del x.xlsx.base64

On my windows machine I can run the following command in cygwin. I assume it works fine in Linux.

base64 -d -i spool_file_name > spreadsheet_name.xlsx

If you fail to included the -i option, it may complain about invalid input. Turns out my cygwin version was not forgiving of the carriage return characters (\015) sqlplus put into the file. It was fine with the linefeeds (\012). The -i option makes it ignore them. It may not be necessary on a true Unix machine or if there is an option to turn off carriage returns in sqlplus on Windows. I didn’t try to find out.

On windows the following command works just fine.

certutil -decode spool_file_name spreadsheet_name.xlsx

In my small example the encoding added around 25% to the file size. I’m not claiming that is the amount you will get, but it should be in that ballpark. It isn’t enough to break the bank.

Conclusion

If you are looking to get a BLOB out of Oracle and you don’t have access to a better scripting language or tool, knowing you can at least do it with sqlplus is a nice option to have in your back pocket.

Sql Tuning for Multiple Use Cases

Sun, 17 Oct 2021 11:30:00 -0400

Introduction

The primary goal of tuning an Oracle SQL statement is to obtain the best performance (shortest response time and minimal resourse utilization) for the intended task. The best answer usually achieves both, but occasionally you sacrifice some of one for the other.

The Oracle optimizer is good at achieving an optimal execution plan (access path, join order, join mechanism), defined as minimal cost, if it has sufficient statistics about the data. There are reams of books and articles about it. I bashed my head against the rocks of Jonathan Lewis’s book Cost Based Oracle Fundamentals for several years. The subject is so massive and fluid, I do not even pretend to get it all. This article is not about that.

The problem to be solved is that we have a single SQL statement with multiple use cases, each of which require a different execution plan.

I will present two use cases for this article.

A nightly batch that touches a full set of data
An incremental, near real-time batch that touches a very small subset of the data

Multiple Use Case SQL Statements

(In practice I have third and forth use cases of nightly “correction” batches that touch what can be either middling sized sets of the data or really huge, almost full sized subsets.)

How can this be addressed?

Adaptive Plan

Modern Oracle databases have dynamic execution plans where the database attempts to re-plan after finding that the statistics used to calculate the original plan do not match reality. That is a good idea, and I am sure it is helpful in situations where the data changes slowly over time. For example at the start of the day one of the source tables may be tiny and a Full Table Scan (FTS) access path is chosen. As the table fills during the day, the adaptive plan logic may determine an Index Access Path is lower cost and switch to it. Adaptive plans may be a great idea and it may solve many issues in many environments,but I have not personally experienced that. Your mileage may vary.

The adaptive plan mechanism does not help sufficiently with this use case. Imagine that we start with the nightly batch case where the driver table statistics would tip off the optimizer to use FTS and Hash join plans. Works great for the nightly batch which runs for thirty minutes (or hours). Now we begin the daily mini-batches. We re-gather statistics on our driver table, but Oracle does not bother to re-evaluate our execution plan. Nope, the idiot savant has a perfectly good plan already, and it is going to use it. It does full table scans and hash joins to obtain what we need for our tiny subset of accounts, wasting resources and taking forever on the first near real-time run. Probably on the second one too. By some later run (perhaps hours later) the adaptive plan technology may figure out something isn’t right and substitute a better plan. By now though we are fielding calls from the business that stuff isn’t working right. Booooooooo!

Even worse would be the behavior that night. Oracle has a good plan now for the small driver table footprint but when we load it up for the nightly batch, Oracle continues to use an unscalable execution plan with index access that runs all night. Uggghhhh!

Invalidate the Plan

We could do something to invalidate the plan forcing Oracle to evaluate the SQL again with the changed statistics.

DBMS_SHARED_POOL

Although there is a package named DBMS_SHARED_POOL that provides fine grained control over invalidating individual sql plans, I have yet to meet a DBA willing to grant execute on the package and I do not blame them one bit. You would also have to do so on every instance of a RAC.

DBMS_STATS

There is also a parameter you can set during statistics gathering that is supposed to invalidate any SQL plan using the table (no_invalidate => FALSE), but it comes with caveats and is generally ignored for at least a while. (DMS_STATS.AUTO_INVALIDATE default value is to let Oracle decide and it does not invalidate the plans immediately.)

DDL Invalidates Plans

At least for Oracle versions 10 through 19, updating a comment on a table counted as DDL that marks all SQL that used the table as needing to be parsed again. (You do have to make the comment different than the existing one though or it is a noop.) Observe:

select * from app_log_app
;
SELECT sql_id,  parse_calls, sql_text
FROM v$sql
WHERE sql_text LIKE '%app_log_app'
AND sql_text NOT LIKE '%v$sql%'
;

SQL_ID	PARSE_CALLS	SQL_TEXT
4t32sfzvq63zf	1	select * from app_log_app

Notcie the number of parse calls is one.

BEGIN
EXECUTE IMMEDIATE q'{COMMENT ON TABLE app_log_app IS 'invalidate plan comment }'
    ||TO_CHAR(SYSDATE,'MM/DD/YYYY HH24:MI:SS')
    ||q'{'}';
END;
/
-- repeat SQL listed above to query the table then v$sql.

SQL_ID	PARSE_CALLS	SQL_TEXT
4t32sfzvq63zf	2	select * from app_log_app

DDL, even a simple COMMENT, invalidates all plans that reference the object, forcing a re-parse. So we could do this once during the nightly batch after gathering statistics on our driver table, and again on the first run of the mini-batch in the morning. It is a bit fragile though as a solution, and there is a downside that there could be large numbers of SQLs that need to be re-parsed if the table you chose was a common one. The scenario I describe it would be a driver table specific to our process, but one should be aware of the possible impact. Nevertheless, this is a viable answer for the multiple Use Case issue assuming we keep gathering statistics on the driver table when needed.

Multiple Sql Statements

So what if we just use two different SQL statements in our code so that we get the right plan for each case?

My response, having gone down that road, is “what if they come up with different answers?”. This is particularly egregious if you wind up only testing one path. Worse yet when you go to make changes, you have to make it in two places.

It may not be possible to completely avoid this though. If you decide to use a pattern that sometimes uses Update/Delete of records in place, and sometimes does a Create Table AS (CTAS) followed by partition exchange, then of course you will be executing different SQL for the different use cases.

Yet whenever possible we should strive to have a single set of code that implements the business logic. This implies we will likely be using Dynamic SQL for the task. Even so, when building the dynamic SQL it would be best if we could avoid having more than one set of code that implements the same logic.

“Sure” you say, you get that. But there will come a time when you do it anyway for expediency. I’ve done that, and have the scars to prove it. If you are like me, you will likely have to learn the hard way once or twice before it sticks. Maybe this article will help you avoid as many scars as this slow learner has accumulated.

It turns out that using two different SQL statements is the answer I’m proposing as far as the database is concerned, just using a common base.

Use Case Specific Hints

Many purists turn up their noses at hints, and lecture you to just make sure your statistics are correct so that the optimizer can determine the best answer.

I can happily engage in a religious war dialogue on the subject if you are one who wishes to be pendantic on either side. I can argue both sides fairly well. I agree that if you can solve the issue by controlling the statistics on the tables and indexes, then by all means do so. In particular this means making rational decisions about histograms and partition level statistics. Mastering that can address many issues with both current and future SQL.

People who argue against hints often have their opinion informed by too often seeing inappropriate hints used by well meaning developers or business users. I get that too. I’m even guilty of it and I’m sure you are too at least at times over your career.

I can also make the case that it is not possible to solve all problems without hints. The case of a piplelined TABLE(f()) function in your query means the optimizer has no clue how many records will come from it. You can give it a cardinality hint, but what if the number of records is variable with the use case?

We also just demonstrated that gathering statistics again on a table does not invalidate existing plans.

There are other scenarios where you as the data expert simply have more information than the optimizer will determine from statistics, regardless of how carefully you gather or even customize the statistics it uses. That rabbit hole is deep.

Let’s embrace the suck and use a minimal set of hints that will guide the optimizer into understanding our use case. The term “minimal” leaves some wiggle room. For the scenario I described we might get by with as little as a single CARDINALITY hint on the driver table plus a degree of parallel hint for the entire query. Example:

    v_sql := q'!INSERT INTO ...
        SELECT /*+ CARDINALITY(d __DRIVER_TABLE_CARDINALITY__) __PARALLEL__ */
            ...
        FROM my_driver_table d
        INNER JOIN ...
    !';
    ...
    EXECUTE IMMEDIATE REPLACE(
        REPLACE(v_sql, '__DRIVER_TABLE_CARDINALITY__'
            ,CASE WHEN p_is_nightly_batch THEN '1000000' ELSE '10' END
            ), '__PARALLEL__', 
                ,CASE WHEN p_is_nightly_batch THEN 'PARALLEL(16)' ELSE 'NO_PARALLEL' END
    );

When we call our procedure the parameter p_is_nightly_batch will be TRUE or FALSE. Now the numbers 10 and a million are not exactly what we have in our driver table, and in fact we could get a mini-batch where the number of records is enough that we would be better off with a different plan than that we get from telling the optimizer to assume 10 records. That is something you might have to experiment with.

Note that this winds up being two different SQL statements in v$sql, each with a different plan.

Putting the parallel degree into the hint is somewhat controversial. One can say it should be determined from the DOP on the table. There are other considerations depending on system load. That is a topic for another day.

In practice I would likely put those values, 10 and 100000 plus the DOP, into a parameter table that I could adjust at run time. (See app_parameter in plsql_utilities.) That way if something happened either over time or during a particular event, I could adjust the values fed to the optimizer on the fly, and since the text of the SQL is different from that already parsed, it of course parses again with the new hint value.

This may not be enough though. There are cases where the optimizer simply refuses to come up with a correct plan without hints, and you must bludgeon it into submission. I offer this article on hash joins as an example of how you might want to compel the optimizer to do things a particular way (though I am almost certain Mr. Lewis would never say the words “bludgeon optimizer into submission”).

You may likely find yourself needing to use more than a single cardinality hint plus degree of parallelism. Example:

    v_sql := q'!INSERT INTO ...
        SELECT 
            /*+ 
                __USE_CASE_HINT__
            */
            ...
        FROM my_driver_table d
        INNER JOIN big_table1 t1
            ON ...
        INNER JOIN midsize_table2 t2
            ON ...
        INNER JOIN big_table t3
            ON ...
    !';
    ...
    EXECUTE IMMEDIATE REPLACE(v_sql, '__USE_CASE_HINT__'
        ,CASE WHEN p_is_nightly_batch 
            THEN q'+
                leading(d t1 t2 t3)
                full(d)
                use_hash(t1) full(t1) swap_join_inputs(t1)
                use_hash(t2) full(t2) no_swap_join_inputs(t2)
                use_hash(t3) full(t3) swap_join_inputs(t3)
                +'
            ELSE q'+
                leading(d t1 t2 t3)
                full(d)
                use_nl(t1) index(t1)
                use_nl(t2) index(t2)
                use_nl(t3) index_ss(t3)
                +'
            END
    );

As you can imagine it can get fairly complex especially if you have a very large query perhaps broken down using WITH query subfactoring (as I heartily recommend). It would be good to establish a naming convention for the hint parameters. You may also want to add an option to the procedure call that prints the SQL with DBMS_OUTPUT rather than executing it. It will help with debugging and establishing the best plan.

Conclusion

When you have one business problem with multiple SQL optimization use cases, make every attempt to keep the business logic in a single set of code. Decorate the dynamic sql with replaceable hints that are appropriate for the use case. Use the minimal set of hints that achieve the objective, but also give yourself some room to adapt when the datbase changes out from under you. Parameterizing the hint values so that they may be changed at run time is a good option that does NOT mean you are changing the code at run time. The actual business logic is that which was tested and promoted to production. Only the optimization is changing and that in response to a business need. This is a solution that comports with best practices while providing some flexibility.

Why do I need Inline PL/SQL Methods?

Sat, 25 Sep 2021 11:30:00 -0400

Declaring PL/SQL in an Oracle SQL WITH Clause

The capability to define PL/SQL functions and procedures inside an Oracle SQL query (and even the query portion of DML statements) was added in Oracle version 12.1. The Oracle documentation for it that I have found so far is sparse. Tim Hall of Oracle Base fame has a good primer on it WITH Clause Enhancements in Oracle Database 12c Release 1 (12.1).

The primary reason (allegedly) that Oracle provided for defining the PL/SQL code inline is to improve performance by avoiding context switching. Yet in the same release we were given PRAGMA UDF which allows optimizing schema level PL/SQL functions as inline. That seems a wash.

The best case I have for deploying inline methods in production code is when the logic is specific to that single query, and is not easily and cleanly done directly in SQL. There are not that many tasks that are so difficult in SQL that this condition exists, but there are some where procedural code is just a better answer. I would also add a caveat that if the procedure is big and gnarly, I would rather compile it in the schema than trying to debug it in the middle of a giant query.

Another Reason for Needing Inline PL/SQL

I have another use for inline PL/SQL. Consider the case where you cannot deploy schema level PL/SQL. The best example I have is doing adhoc queries on a production system. You can deploy PL/SQL to that system after going through the formal process, but not today.

Maybe you have a real need for PL/SQL, like calling procedures with OUT parameters then selecting the value in a query, but you have query-only access on a system. Or perhaps your access to the database elements you need is through roles, not direct grants, so you cannot deploy PL/SQL that uses those elements.

Role Access for Inline PL/SQL

For this test my schema has not been granted SELECT on HR.job_history. I have the role privilege HR_SELECT, and that role has been granted SELECT on HR.job_history. If I attempted to create a function that read from the table HR.job_history, the create would fail. The inline PL/SQL function works though.

WITH FUNCTION get_hist_job_id(p_emp number) RETURN VARCHAR2
AS
    l_job_id VARCHAR2(30);
BEGIN
    SELECT MAX(job_id) KEEP (DENSE_RANK FIRST ORDER BY end_date DESC) INTO l_job_id
    FROM hr.job_history
    WHERE employee_id = p_emp
    ;
    RETURN l_job_id;
END;
SELECT get_hist_job_id(176) FROM dual;
/

GET_HIST_JOB_ID(176)
--------------------------------------------------------------------------------
SA_MAN

Granted that the function doesn’t do anything that we could not have done directly in the query, but the test was merely to prove that we can avoid the issue with grants through roles versus direct grants to the schema owner. If you need PL/SQL and face a limitation where you cannot get the direct grants, this might be a way around it. Note that this is not likely to please certain control freaks, so keep it to yourself.

You might also be able to get around this limitation by using dynamic sql (EXECUTE IMMEDIATE) and AUTHID CURRENT_USER. I admit to not having a complete grasp of all of the intricacies of the privilege stack where dynamic sql is involved, but I think it can be done. I like the inline PL/SQL method for it though.

Selecting a Value from a Procedure OUT Parameter

Consider a scenario where the only methods available to you for retrieving some value is through a procedure with an OUT parameter. You cannot call that directly from a SQL select. In this system we cannot deploy schema level PL/SQL today, so we will use an inline function to provide the result.

The DBMS_LOB package has a procedure LOADBLOBFROMFILE that puts the result in an OUT parameter. I want to SELECT that BLOB value and save it to my local drive (which I can do easily from Toad or SqlDeveloper).

As of Oracle 12.2 you can do this in straight SQL (see TO_CLOB and TO_BLOB enhancements), but I’m going to use the example anyway.

WITH FUNCTION get_blob(p_directory VARCHAR2, p_filename VARCHAR2) RETURN BLOB
AS
    l_bfile         BFILE;
    l_blob          BLOB;
    l_blob_out      BLOB;
    l_dest_offset   INTEGER := 1;
    l_src_offset    INTEGER := 1;
BEGIN
    l_bfile := BFILENAME(p_directory, p_filename);
    DBMS_LOB.fileopen(l_bfile, DBMS_LOB.file_readonly);
    IF DBMS_LOB.getlength(l_bfile) > 0 THEN
        DBMS_LOB.createtemporary(l_blob,TRUE);
        DBMS_LOB.loadblobfromfile(
            dest_lob        => l_blob
            ,src_bfile      => l_bfile
            ,amount         => DBMS_LOB.getlength(l_bfile)
            ,dest_offset    => l_dest_offset
            ,src_offset     => l_src_offset
        );
        l_blob_out := l_blob;
        DBMS_LOB.freetemporary(l_blob);
    END IF;
    DBMS_LOB.fileclose(l_bfile);
    RETURN l_blob_out;
END;
SELECT get_blob('TMP_DIR','x.xlsx') FROM dual
;
/

That gives me a “(BLOB)” in the query result in SqlDeveloper that I can download to my local disk.

Procedure Required to Produce Result

In a similar vein you may require execution of multiple PL/SQL procedural steps to produce the output you ultimately want to select in your adhoc query.

Marc Bleron has an excellent tool for generating MS Excel files directly in the database ExcelGen - An Oracle PL/SQL Generator for MS Excel Files. Most of the time you will use this tool within PL/SQL procedures or packages that you deploy. Yet there are times when it can be useful for an adhoc task. Sure, you can have Toad or Sqldeveloper produce CSV or even Excel files from a query, but they do not match this for capability. And if you happen to have already written a procedure that does much of what you need for your adhoc query, you can quickly copy/paste the code into a WITH clause and be ready to rock and roll.

WITH 
FUNCTION get_xlsx(p_src SYS_REFCURSOR) RETURN BLOB AS
    v_blob          BLOB;
    v_ctxId         ExcelGen.ctxHandle;
    v_sheetHandle   BINARY_INTEGER;
BEGIN
        v_ctxId := ExcelGen.createContext();
        v_sheetHandle := ExcelGen.addSheetFromCursor(v_ctxId, 'Employee Salaries', p_src, p_sheetIndex => 1);
        -- freeze the top row with the column headers
        ExcelGen.setHeader(v_ctxId, v_sheetHandle, p_frozen => TRUE);
        -- style with alternating colors on each row. 
        ExcelGen.setTableFormat(v_ctxId, v_sheetHandle, 'TableStyleLight2');
        -- single column format on the salary column. The ID column keeps default format
        ExcelGen.setColumnFormat(
            p_ctxId     => v_ctxId
            ,p_sheetId  => v_sheetHandle
            ,p_columnId => 5        -- the salary column
            ,p_format   => '$#,##0.00'
        );
        v_blob := ExcelGen.getFileContent(v_ctxId);
        ExcelGen.closeContext(v_ctxId);
        RETURN v_blob;
END;
-- begin sql portion 
add_bilbo AS (
    SELECT e.employee_id AS employee_id, e.last_name, e.first_name, d.department_name, e.salary
    FROM hr.employees e
    INNER JOIN hr.departments d
        ON d.department_id = e.department_id
    UNION ALL
    SELECT 999 AS employee_id, 'Baggins' As last_name, 'Bilbo' as first_name, 'Sales' AS department_name
        ,123.45 AS salary
    FROM dual
), emp_curs AS (
    SELECT * FROM add_bilbo ORDER BY last_name, first_name
) SELECT get_xlsx(CURSOR(SELECT * FROM emp_curs)) FROM DUAL
;
/

The trick of passing a CURSOR variable using the last named subquery of the WITH clause (last line of above sample) is awesome. Many times I’ve written huge queries that were stuffed into a CURSOR cast to pass into a Function before I discovered this handy syntax.

Since the ExcelGen package uses session level package global variables to store everything, you could probably execute an anonymous PL/SQL block using a client declared context variable that does the first part of this through setColumnFormat, then use a simple

SELECT ExcelGen.getFileContent(:v_ctxId) FROM dual;

to get the result. I like the WITH PL/SQL version much better.

Here is another example using PdfGen, a package I wrote that wraps the core as_pdf3 written by Anton Scheffer.

WITH FUNCTION get_blob(p_src SYS_REFCURSOR) RETURN BLOB
IS
    v_blob      BLOB;
    v_widths    PdfGen.t_col_widths;
    v_headers   PdfGen.t_col_headers;
    v_formats   PdfGen.t_col_headers;
BEGIN
                                                -- Similar to the sqlplus COLUMN HEADING commands
    v_headers(1) := 'Employee ID';
    v_widths(1)  := 11;
    v_headers(2) := 'Last Name';
    v_widths(2)  := 25;
    v_headers(3) := 'First Name';
    v_widths(3)  := 20;
                                                -- will not print this column, 
                                                -- just capture it for column page break
    v_headers(4) := NULL;                       --'Department Name'
    v_widths(4)  := 0;                          -- sqlplus COLUMN NOPRINT 
    v_headers(5) := 'Salary';
    v_widths(5)  := 16;
    -- override default number format for this column
    v_formats(5) := '$999,999,999.99';
    --
    PdfGen.init;
    PdfGen.set_page_format(
        p_format            => 'LETTER' 
        ,p_orientation      => 'PORTRAIT'
        ,p_top_margin       => 1
        ,p_bottom_margin    => 1
        ,p_left_margin      => 0.75
        ,p_right_margin     => 0.75
    );
    PdfGen.set_footer;                          -- 'Page #PAGE_NR# of "PAGE_COUNT#' is the default
                                                -- sqlplus TITLE command
    PdfGen.set_page_header(
        p_txt_center        => 'Employee Salary Report'
        ,p_font_family      => 'helvetica'
        ,p_style            => 'b'
        ,p_fontsize_pt      => 16
        ,p_txt_left_2       => 'Department: !PAGE_VAL#'
        ,p_font_family_2    => 'helvetica'
        ,p_fontsize_pt_2    => 12
        ,p_style_2          => 'n'
    );
    --
    as_pdf3.set_font('courier', 'n', 10);
    PdfGen.refcursor2table(
        p_src                       => p_src
        ,p_widths                   => v_widths
        ,p_headers                  => v_headers
        ,p_formats                  => v_formats
        ,p_bold_headers             => TRUE     -- also light gray background on headers
        ,p_char_widths_conversion   => TRUE
        ,p_break_col                => 4        -- sqlplus BREAK ON column
        ,p_grid_lines               => FALSE
    );
    v_blob := PdfGen.get_pdf;
    RETURN v_blob;                              
END;
-- begin sql
a AS (
    SELECT e.employee_id, e.last_name, e.first_name, d.department_name
        ,SUM(salary) AS salary          -- emulate sqplus COMPUTE SUM
    FROM hr.employees e
    INNER JOIN hr.departments d
        ON d.department_id = e.department_id
    GROUP BY GROUPING SETS (
                                        -- seemingly useless SUM on single record, 
                                        -- but required to get detail records
                                        -- in same query as the subtotal and total aggregates
        (e.employee_id, e.last_name, e.first_name, d.department_name)
        ,(d.department_name)            -- sqlplus COMPUTE SUM of salary ON department_name
        ,()                             -- sqlplus COMPUTE SUM of salary ON report - the grand total
    )
), b AS (
    SELECT employee_id
        -- NULL last_name indicates an aggregate result.
        -- NULL department_name indicates it was the grand total
        -- Similar to the LABEL on COMPUTE SUM
        ,NVL(last_name, CASE WHEN department_name IS NULL
                            THEN LPAD('GRAND TOTAL:',25)
                            ELSE LPAD('DEPT TOTAL:',25)
                        END
        ) AS last_name
        ,first_name
        ,department_name
        ,salary
    FROM a
    ORDER BY department_name NULLS LAST     -- to get the aggregates after detail
        ,a.last_name NULLS LAST             -- notice based on FROM column value, not the one we munged in resultset
        ,first_name
)
SELECT get_blob(CURSOR(SELECT * FROM b)) FROM dual
;
/

An example page from that pdf report follows. That is all a bit much for something you are doing adhoc, but given the scrutiny placed over moving code into production in modern corporate environments, you may find yourself reaching at times. This is a capability that will allow you to reach pretty far without deploying any code to production. That is nice to have in your back pocket when everyone is looking to you for a miracle.

Writes

Since this facility is part of a SELECT query (or the SELECT portion of DML), it is limited to read only operations. If your inline procedure or function will perform DML or DDL, you need to declare it with PRAGMA AUTONOMOUS_TRANSACTION as mentioned in this asktom question.

I won’t disagree with the assertion that it is generally a bad idea, but when constrained by circumstances, one does as needs must.

Conclusion

Although avoidance of context switching may have been the driving force behind Oracle delivering the inline PL/SQL capability for the SQL engine, we can take advantage of it for many reasons. I hope these examples were enlightening.

The Ubiquitous CSV File

Fri, 10 Sep 2021 11:30:00 -0400

The Oh So Common CSV Files

One of the most common methods for transferring data between applications and businesses is the comma separated value format file. It has many advantages:

It is relatively simple.
It is entirely text, so it can be transported between computers running pretty much any operating system.
It can be opened by common desktop applications like Excel.
It isn’t that difficult to generate manually.
Many applications can generate and consume it automatically.

There is a semi-official standard for the format at RFC4180, but if you read it you will find it is fairly permissive about how you can quote fields that contain separators and even newlines.

Parsing a CSV file seems simple until you read all of the rules and sit down to try to do it. I have what I think is a complete solution for parsing CSV in the Oracle PL/SQL function split. The regular expression it employs is so ugly the comments outweigh the code ten to one. Yet it seems to work just fine, passing all of the test cases I can conjour to throw at it.

Loading CSV File

sqlldr (and by extension external tables) support a text import option that should work for most honest CSV input files:

FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'

I did not go out of my way to torture it the way I did my split function, yet the only issues I’ve ever encountered with it had to do with funky non-ascii characters, which is only tangentially related to the CSV format in that it depends on an understood common or convertible character set. When the input file does not comply, as for instance when a non-quoted string contains the delimiter, all bets are off.

If the input file follows RFC4180, I suspect Oracle sqlldr will parse it just fine. I could be wrong.

Generating CSV

From an Oracle database there are tools we can use to generate a CSV output file.

GUI Clients

Toad and Sqldeveloper both have CSV format file output options. You chan choose whether or not to quote and whether or not to include column headers. These work well, but they require a person to execute them. That is not a viable solution to something you want to automate.

SQLcl (aka sql command line)

SQLcl (SQL Developer Command Line) is a Java-based tool. It supports most of the cruft of sqlplus while also adding new stuff and many of the features of SqlDeveloper. Although it ships with the Oracle client, it may or may not be installed on your ETL server. It has a robust set of output formats and is quite handy with CSV, both for exporting and for importing!

I have not seen it used as a replacement for sqlplus and am not completely sure why. It may be that the people who install and maintain database and ETL servers are just dinosaurs who don’t trust new stuff. I mean it is only up to version 20.4. sqlplus on the other hand has been around as long as Oracle has been and it is everywhere.

sqlplus

SQL*Plus 12.2 has added a set markup csv option, and when you turn on text quoting

set markup csv on quote on

it will “escape” double quotes embedded witin the text. This can also give you the column headers in CSV format when you have “set heading on”, so it may be a decent option. We will come back to that.

If you are stuck with an older version, you can concatenate data elements converted to text with comma’s between them in your SQL statement and spool it to a text file. For example:

set linesize 200
set pagesize 0
set heading off
set trimspool on
set echo off
set termout off
spool test1.csv
SELECT 
    TO_CHAR(department_id)
    ||','|| department_name
    ||','|| TO_CHAR(manager_id)
    ||','|| TO_CHAR(location_id)
FROM hr.departments 
ORDER BY department_name
;
spool off

I have seen countless scripts like this in my career. I cringe every time I see one because I know it is one careless data entry event away from waking up some poor support person in the middle of the night when the file won’t load because there is a comma in one of the field values. A little better pattern can protect against that:

set linesize 200
set pagesize 0
set heading off
set trimspool on
set echo off
set termout off
spool test2.csv
SELECT 
                  TO_CHAR(department_id)
    ||','|| '"'|| department_name           ||'"'
    ||','||       TO_CHAR(manager_id)
    ||','||       TO_CHAR(location_id)
FROM hr.departments 
ORDER BY department_name
;
spool off

Aside from how ugly it is and the chance that you make a mistake while typing it in, there is another potential issue. What if the comma we fear is not what gets entered into the field value? What if it is a double quote mark? What if it is a newline character?

I chose to not enclose the numeric fields with double quotes, but what if some joker changed the NLS settings on us and the new default number format uses a comma for the decimal? DOH!

Another issue with this is that you don’t get the column headers. If you need them there are several clever ways to do it. You can google search it. I’m not happy with any of them.

The markup csv option available starting in sqlplus that comes with Oracle 12c looks promising. Let’s try it out. We’ll add some tricks with the NLS date and numeric default formats to see how smart it is. I’ve made both default conversion formats contain a comma in the result.

First we will try with quote off.

ALTER SESSION SET nls_date_format = 'YYYYMMDD, HH24:MI';
ALTER SESSION SET nls_numeric_characters = ',.';
set markup csv on quote off
set linesize 200
set pagesize 0
--set heading off
set trimspool on
set echo off
set termout off
spool test_markup_noquote.csv
SELECT 
    'a string with comma(,)' AS "s1"
    ,'a string with dquote(")' AS "s2"
    ,'a string with newline(
)' AS "s3"
    ,SYSDATE AS "date1"
    ,12345 / 100 AS "num1"
FROM DUAL
;
spool off

And now the spool file:

$ cat test_markup_noquote.csv

s1,s2,s3,date1,num1
a string with comma(,),a string with dquote("),a string with newline(
),20210910, 04:52,123,45

As I suspected if you set quote off in the markup directive, you get what you paid for. That does not comply with the CSV format specification.

It also seems to have both a leading and trailing blank line in it. Not sure what is up with that or whether you can get rid of them short of doing something in the shell afterward. If I decide to use it for a job, I’ll have to figure that out.

Let’s try the same but with quote on

ALTER SESSION SET nls_date_format = 'YYYYMMDD, HH24:MI';
ALTER SESSION SET nls_numeric_characters = ',.';
set markup csv on quote on
set linesize 200
set pagesize 0
--set heading off
set trimspool on
set echo off
set termout off
spool test_markup_wquote.csv
SELECT 
    'a string with comma(,)' AS "s1"
    ,'a string with dquote(")' AS "s2"
    ,'a string with newline(
)' AS "s3"
    ,SYSDATE AS "date1"
    ,12345 / 100 AS "num1"
FROM DUAL
;
spool off

And now the spool file:

"s1","s2","s3","date1","num1"
"a string with comma(,)","a string with dquote("")","a string with newline(
)","20210910, 05:06",123,45

Well that is interesting. It handles the character types that contain double quotes just like the RFC specifies by doubling up the internal doublequote marks. A competent CSV parser should have no trouble with that. As a bonus, it was smart enough to put double quotes around the date value converted by the default NLS format. Sweet! But it did NOT quote the number “num1” value even though it contained a separator comma, and that is a potential issue. It is not a very likely issue to happen to you in the real world though. And if you think it could be, you can always do the TO_CHAR conversion of numbers to strings in your query.

Overall using sqlplus with the markup csv option looks to be a fine way to generate CSV files.

Custom Tools

Maybe the ability to run a batch operation on an ETL server using sqlplus is not part of your operational toolset. Maybe you need to generate the file from inside a program that is not able to launch sqlplus, say for example you have a job that is running from DBMS_SCHEDULER_JOBS directly in the database, or the task is launched from a trigger or a web application interacting with the database.

For those scenarios where you cannot use a client program to generate the file, we can generate the file using PL/SQL.

The Hard Way

You can open a file on the database server and manually construct and write the CSV file. This is a fairly common technique I have seen deployed by clients (though they rarely bothered with the bulk collect).

DECLARE
    CURSOR c1 IS
        SELECT e.employee_id, e.last_name, e.first_name, d.department_name, salary
        FROM hr.employees e
        INNER JOIN hr.departments d
            ON d.department_id = e.department_id
        ORDER BY d.department_name, e.last_name, e.first_name
    ;
    SUBTYPE t_c1_rec IS c1%ROWTYPE;
    TYPE t_arr_c1_rec IS TABLE OF t_c1_rec;
    v_arr_c1_rec    t_arr_c1_rec;
    v_file          UTL_FILE.file_type;
    v_dir           CONSTANT VARCHAR2(30) := 'TMP_DIR';
    v_file_name     CONSTANT VARCHAR2(30) := 'test_csv_dbfile.csv';
BEGIN
    v_file := UTL_FILE.fopen(
        filename        => v_file_name
        ,location       => v_dir
        ,open_mode      => 'w'
        ,max_linesize   => 32767
    );
    UTL_FILE.put_line(v_file, 'employee_id,last_name,first_name,department_name,salary');
    OPEN c1;
    LOOP
        FETCH c1 BULK COLLECT INTO v_arr_c1_rec LIMIT 100;
        EXIT WHEN v_arr_c1_rec.COUNT = 0;
        FOR i IN 1..v_arr_c1_rec.COUNT
        LOOP
            UTL_FILE.put_line(v_file,
                TO_CHAR(v_arr_c1_rec(i).employee_id)
                ||',"'||v_arr_c1_rec(i).last_name           ||'"'
                ||',"'||v_arr_c1_rec(i).first_name          ||'"'
                ||',"'||v_arr_c1_rec(i).department_name     ||'"'
                ||',"'||LTRIM(TO_CHAR(v_arr_c1_rec(i).salary,'$999,999,999.99'))
                                                            ||'"'
            );
        END LOOP;
    END LOOP;
    CLOSE c1;
    UTL_FILE.fclose(v_file);
    RETURN;
EXCEPTION WHEN OTHERS THEN
    IF UTL_FILE.is_open(v_file)
        THEN UTL_FILE.fclose(v_file);
    END IF;
    RAISE;
END;
/

lee@linux2 [/tmp]
$ head -10 test_csv_dbfile.csv
employee_id,last_name,first_name,department_name,salary
206,"Gietz","William","Accounting","$8,300.00"
205,"Higgins","Shelley","Accounting","$12,008.00"
200,"Whalen","Jennifer","Administration","$4,400.00"
102,"De Haan","Lex","Executive","$17,000.00"
100,"King","Steven","Executive","$24,000.00"
101,"Kochhar","Neena","Executive","$17,000.00"
110,"Chen","John","Finance","$8,200.00"
109,"Faviet","Daniel","Finance","$9,000.00"
108,"Greenberg","Nancy","Finance","$12,008.00"

That works well enough, but man oh man is it painful.

refcursor-to-csv

William Robertson published a nice package that can generate CSV data that complies with the RFC from any ref cursor. refcursor-to-csv is an approach that is completely within the database rather than in a client. This means you can select the resulting CSV rows in any client or write them to a file on the database server. It is a fine tool that may well meet your needs. He adds some bells and whistles for producing optional header and trailer records that may be exactly what you are looking for.

Here are two examples from Mr. Robertson’s documentation at the above link:

select column_value
from   table(csv.report(cursor(
        select * from dept
    ), p_separator => '|', p_label => 'DEPT', p_heading => 'Y', p_rowcount => 'Y'));

COLUMN_VALUE
----------------------------------------------
HEADING|DEPT|DEPTNO|DNAME|LOC
DEPT|10|ACCOUNTING|NEW YORK
DEPT|20|RESEARCH|DALLAS
DEPT|30|SALES|CHICAGO
DEPT|40|OPERATIONS|BOSTON
ROW_COUNT|DEPT|4

6 rows selected.

And the example of writing to a file on the database server:

declare
    l_dataset sys_refcursor;
begin
    open l_dataset for select * from dept;

    csv.write_file
    ( p_dataset => l_dataset
    , p_separator => '|', p_label => 'DEPT', p_heading => 'Y', p_rowcount => 'Y'
    , p_directory => 'DATA_FILE_DIR'
    , p_filename => 'dept.csv' );
end;

app_csv

I was inspired by Mr. Robertson’s work to extend it in several ways where I wanted more flexibility. I also had another use for the underlying mechanism of using DBMS_SQL to gather the cursor results for any unknown query. I created base Object types I call app_dbms_sql and then built app_csv on top of those.

The app_csv documentation on use cases show selecting from a TABLE function, writing to a database server file, and retrieving a CLOB. The latter is particularly useful if you want to email the CSV data as an attachment. (See html_email for a tool that can send email with attachments from the database server.)

Both of these tools process completely within the database engine.

ExcelGen

If the only reason you are producing CSV files is because you want to give them to a user to open in Excel, why stop there? ExcelGen is a package by Marc Bleron that can convert a refcursor (just like app_csv and refcursor-to-csv do) and generate an XLSX file. Maybe your users are happy enough with opening a CSV file in excel and mucking with the column formats for a few minutes every day, but you can go the extra step without a whole lot more trouble. Plus as a bonus if you have multiple queries and files you are producing for them, you can combine them as separate tabs/sheets in the same workbook. The format is compressed too so files are smaller.

It is just a little bit more work than producing CSV files, yet the end product is sure to be appreciated by your users.

Conclusion

These are some techniques you may be able to use to fit a need. Hope it was helpful.

Oracle Object Types as Application Interface

Thu, 09 Sep 2021 11:30:00 -0400

Scenario

As the Oracle specialist on an application development team your goal is to provide an interface to the application data for use by team members developing in other tools such as a Java or .NET development platform.

Object Type Application Interface Deployment Diagram

Traditional Method

You provide queries and perhaps views and/or materialized views for use by the team. Perhaps you provide functions or procedures that return cursors or maybe a Pipelined Table function. These are all perfectly valid methods for providing a two dimensional data structure of rows and columns to the application. You can build business rules into the queries and methods you provide, and assist with database operations where you add value to the team.

What if your data is more complex? When the data is logically represented by parent/child relationships with multiple child objects, the traditional method might simply duplicate common parent data on every row. The database client, a Java program for example, will likely build a method to consume this data, pulling the common data elements into one structure and the child data elements into another, rebuilding the logical relationship into memory. This is wasteful on multiple levels. There is bandwidth and memory waste with transferring the redundant data as well as the development time waste of recreating the structure after it had been flattened during the transfer.

Remember also that in a high volume environment, each round trip to the database has a real cost in time and resources. Most modern Web based applications are built using stateless, single operation connections from a pool. This is relatively efficient, but there is real overhead in time and resources for each separate database call. Whenever practical, it is better to retrieve necessary data in a single call rather than multiple round trips to the database (assuming memory is not the constraint).

Using Oracle Object Types

Modern Oracle database connectors provide classes and methods for consuming complex structures from the database in the form of database Types. These can be traditional record Types as well as complex Objects, which includes Collections. The Oracle drivers will transport the complex structures over the wire efficiently and convert them into native data structures. IDE’s usually have tools to see these object types when you point them at a database procedure, and assist with building custom structures and classes for consuming and accessing the data.

Object Hiearchy Example

Although Object types can have methods, we will focus on using them as a hierachical structure. The MAP method is defined in this example to make it convenient to sort the objects as they are collected. You may be able to use other techniques to sort before aggregating, but this is the recommended way. Don’t focus on that too much. The main point is that the Objects are basically Structures for our purpose.

We will use the HR sample schema for the demonstration. The example is contrived in that the overhead of putting the department data into a flattened row or running two separate queries, is minimal, and we would not bother normally. Pretend that we have a much larger common object, perhaps multiple disparate child arrays, and/or extremely high volume that will justify the added complexity.

CREATE OR REPLACE TYPE employee_udt AS OBJECT (
     last_name          VARCHAR2(25)
    ,first_name         VARCHAR2(20)
    ,email              VARCHAR2(25)
    ,phone_number       VARCHAR2(20)
    ,hire_date          DATE
    ,salary             NUMBER(8,2)
    -- for sorting
    ,MAP MEMBER FUNCTION full_name RETURN VARCHAR2
);
/
CREATE OR REPLACE TYPE BODY employee_udt AS
    MAP MEMBER FUNCTION full_name RETURN VARCHAR2
    IS
    BEGIN
        RETURN last_name||', '||first_name;
    END;
END;
/
CREATE OR REPLACE TYPE arr_employee_udt AS TABLE OF employee_udt;
/
CREATE OR REPLACE TYPE department_udt AS OBJECT (
     department_name    VARCHAR2(30)
    ,manager_last_name  VARCHAR2(25)
    ,manager_first_name VARCHAR2(20)
    ,arr_employee       arr_employee_udt
    ,MAP MEMBER FUNCTION dmap RETURN VARCHAR2
);
/
CREATE OR REPLACE TYPE BODY department_udt AS 
    MAP MEMBER FUNCTION dmap RETURN VARCHAR2
    IS
    BEGIN
        RETURN department_name;
    END;
END;
/
CREATE OR REPLACE TYPE arr_department_udt AS TABLE OF department_udt;
/

Each department object will contain all of the employees in the department. The top level array of departments object is a single unit that can be transferred in one database call. Alternatively, if that object could be too large to be efficient, a cursor can return rows of department_udt objects, perhaps bulk fetched depending on how it is called. Here is a package that provides both. The query is written in multiple pieces using WITH subquery factoring to make it easier to follow (a practice I recommend to keep large queries easier to understand).

CREATE OR REPLACE PACKAGE employees_by_dept_pkg AS
    PROCEDURE get(p_arr_department OUT arr_department_udt);
    FUNCTION get RETURN arr_department_udt PIPELINED;
END;
/
CREATE OR REPLACE PACKAGE BODY employees_by_dept_pkg AS
    CURSOR g_c IS
        WITH d AS (
            SELECT hd.department_id, hd.department_name, he.last_name AS manager_last_name, he.first_name AS manager_first_name
            FROM hr.departments hd
            INNER JOIN hr.employees he
                ON he.employee_id = hd.manager_id
        ), e AS (
            SELECT department_id
                ,employee_udt(
                    last_name       => e.last_name
                    ,first_name     => e.first_name
                    ,email          => e.email
                    ,phone_number   => e.phone_number
                    ,hire_date      => e.hire_date
                    ,salary         => e.salary
                ) AS emp
            FROM hr.employees e
        ), e2 AS (
            SELECT department_id
                -- uses employee_udt.map method to sort
                ,CAST(COLLECT(emp ORDER BY emp) AS arr_employee_udt) AS arr_employee
            FROM e
            GROUP BY department_id
        ), d2 AS (
            SELECT department_udt(
                     department_name    => d.department_name
                    ,manager_last_name  => d.manager_last_name
                    ,manager_first_name => d.manager_first_name
                    ,arr_employee       => e2.arr_employee
                ) AS dept
            FROM d
            INNER JOIN e2
                ON e2.department_id = d.department_id
        ) SELECT dept
        FROM d2
        ORDER BY dept
        ;

    PROCEDURE get(
        p_arr_department    OUT arr_department_udt
    ) AS
    BEGIN
        LOOP
            -- if the cursor was previsously opened and not yet closed in this session
            -- then close it and try again. 
            BEGIN
                OPEN g_c;
                EXIT;
            EXCEPTION WHEN CURSOR_ALREADY_OPEN
                THEN CLOSE g_c;
            END;
        END LOOP;
        FETCH g_c BULK COLLECT INTO p_arr_department;
        CLOSE g_c;
    END;

    FUNCTION get RETURN arr_department_udt PIPELINED
    AS
        v_arr_department    arr_department_udt;
    BEGIN
        LOOP
            -- if the cursor was previsously opened and not yet closed in this session
            -- then close it and try again. 
            BEGIN
                OPEN g_c;
                EXIT;
            EXCEPTION WHEN CURSOR_ALREADY_OPEN
                THEN CLOSE g_c;
            END;
        END LOOP;

        LOOP
            -- pretending object is very large, thus small limit
            FETCH g_c BULK COLLECT INTO v_arr_department LIMIT 20;
            EXIT WHEN v_arr_department.COUNT = 0;
            FOR i IN 1..v_arr_department.COUNT
            LOOP
                PIPE ROW(v_arr_department(i));
            END LOOP;
        END LOOP;
        CLOSE g_c;
        RETURN;
    END;
END employees_by_dept_pkg;
/

To call the procedure version from a language using an Oracle database driver you would typically find the package in your IDE interface and allow it to help you build a class to call the procedure and populate an internal structure via the OUT parameter. For us mere mortals restricted to standard database tools such as sqldeveloper, we can use the pipelined table function

SELECT *
FROM TABLE(employees_by_dept_pkg.get);

Here is one row as represented in sqldeveloper query result window:

DEPARTMENT_NAME	MANAGER_LAST_NAME	MANAGER_FIRST_NAME	ARR_EMPLOYEE
Accounting	Higgins	Shelley	LEE.ARR_EMPLOYEE_UDT(LEE.EMPLOYEE_UDT(‘Gietz’, ‘William’, ‘WGIETZ’, ‘515.123.8181’, ‘2002-06-07 00:00:00.0’, 8300), LEE.EMPLOYEE_UDT(‘Higgins’, ‘Shelley’, ‘SHIGGINS’, ‘515.123.8080’, ‘2002-06-07 00:00:00.0’, 12008))

A query that joins back to the employee object column flattens it back out to one record per employee with duplication of the department information, but at least we can see everything.

SELECT department_name, manager_last_name, manager_first_name, e.*
FROM TABLE(employees_by_dept_pkg.get) d
INNER JOIN TABLE(d.arr_employee) e ON 1=1
WHERE rownum <= 10
;

DEPARTMENT_NAME	MANAGER_LAST_NAME	MANAGER_FIRST_NAME	LAST_NAME	FIRST_NAME	EMAIL	PHONE_NUMBER	HIRE_DATE	SALARY
Accounting	Higgins	Shelley	Gietz	William	WGIETZ	515.123.8181	07-JUN-02	8300
Accounting	Higgins	Shelley	Higgins	Shelley	SHIGGINS	515.123.8080	07-JUN-02	12008
Administration	Whalen	Jennifer	Whalen	Jennifer	JWHALEN	515.123.4444	17-SEP-03	4400
Executive	King	Steven	De Haan	Lex	LDEHAAN	515.123.4569	13-JAN-01	17000
Executive	King	Steven	King	Steven	SKING	515.123.4567	17-JUN-03	24000
Executive	King	Steven	Kochhar	Neena	NKOCHHAR	515.123.4568	21-SEP-05	17000
Finance	Greenberg	Nancy	Chen	John	JCHEN	515.124.4269	28-SEP-05	8200
Finance	Greenberg	Nancy	Faviet	Daniel	DFAVIET	515.124.4169	16-AUG-02	9000
Finance	Greenberg	Nancy	Greenberg	Nancy	NGREENBE	515.124.4569	17-AUG-02	12008
Finance	Greenberg	Nancy	Popp	Luis	LPOPP	515.124.4567	07-DEC-07	6900

Conclusion

The technique of using Oracle Object Types to represent complex data can be an efficient way to provide applications what they need in a format that is well suited. It allows the back-end database developer the opportunity to shelter the human interface developer from needing to understand the full database model. Instead they can focus on the data model presented to them at the application level. This also enhances the ability to put business rules into the database back end as opposed to (or in addition to) building that logic on the front end. That may or may not be a desired approach for your organization, but at a minimum the business logic that is necessary at the database level can be enforced.

Install Oracle Enterprise Linux and Oracle Database on a Dual Boot Windows 10 Machine

Sat, 09 May 2020 01:00:00 -0400

Project

Install Oracle Enterprise Linux and Oracle Database 19c on a dual boot Windows 10 machine.

This will be a full Oracle install including ASM and grid for my professional experience rather than a minimal database under a virtual box.

Note that I do not have a service contract with Oracle, am not paying $500 for one for my personal use, and therefore cannot get to the support documentation or the Oracle Unbreakable Linux Network (ULN). This includes all of the bug fixes and reports and lots of helpful information. If you run into trouble without this you are SOL. I learned the hard way to stick with the supported releases rather than go off trying to figure it out. I spent a week with Ubuntu and then another few days with Oracle Enterprise Linux 8 latest and greatest (which is not supported for Oracle 19c!!!). Just don’t do it. Frustration will be immense if you cannot get to the support documentation. [Note: You can get the database to install, but grid is another matter].

This is our Home Theatre PC. Other than acting as a file server, the main usage it gets is Netflix via a web browser. That will work just find under Linux too, so I’m hopeful wife won’t mind which OS is running most of the time. That is the goal.

Credits

I found all sorts of references using google searches and learned from them (as well as having been doing similar stuff for 35 years including sysadmin a very long time ago). Unfortunately, I did not save links except in a few cases where I included them. Rest assured though that if you google any of these subjects you will hit the same resources that I worked my way through. I wish I had done a better job with it. Nobody can know all this stuff. I mostly just fumbled my way through.

Hardware

Intel NUC 7i5BNH with 512GB SSD with windows 10 installed.
LaCie d2 Professional SCSI Disk Device 4TB Thunderbolt USB drive ($200!!!!). It isn’t SSD but it is an enterprise level 4TB disk drive that is plenty fast through the USB-C Thunderbolt interface. I’ll be installing Oracle Linux and Oracle database on this drive.
“My Book” 3TB USB drive for windows public smb share for photos. This drive is shared by the windows boot as our “G” drive and I’ll be making the Linux boot share it the same way. The clients use the IP address to mount it and don’t care whether the machine is booted as Windows or Linux.
Although you can shrink the space on main windows drive and install Linux, I’m installing everything on the 4TB Lacie Thunderbolt disk. I’m leaving the main windows drive completely alone, though the boot loader may get touched along the way. My expectation is I can recover completely to the windows normal boot if needed.

Partitioning the Drive

Although you can let the Oracle Linux installer do this, it doesn’t behave the way I want. I want a raw partition to use with ASM for the Oracle data disks. I couldn’t figure out how to make the installer create that partition without putting a file system on it, so I would have to come back and do it later anyway. I found it easier just to use the Windows “Disk Management” tool to do it (or you can boot a gparted image from USB, but I didn’t find it necessary). On windows hit Start then type in “Disk Management”. I decided on the following:

200 MB (198) partition for EFI /boot/efi
1 GB partition for /boot
552 GB partition for Volume Manager

I wanted to leave room for the volume manager partition to grow so I created a partition with 1124.81 GB (the remainder) that I later removed leaving a gap. I realize the volume manager probably could have dealt with it just fine with it split across the disk, but I haven’t explored it yet.

I was going to create a 2TB partition for ASM, but ran into a snag (it was slightly over). The documentation says the 2TB limitation is fixed after Oracle 11, but it is apparently still a problem. Breaking it into two 1TB partitions works fine and also gives me more chance to play with ASM. So, two 1024 GB (1048576 MB) partitions after the gap to use with ASM.

After creating the two partitions for ASM, I deleted the place holder for the volume manager partition growth (1124.81 GB now unallocated). The resulting allocation looks like this (DISK 2 which is the Lacie drive):

BIOS Setup

While in Windows you may want to Turn off Fast Startup. If you google it you will find plenty of entries on how to do it. I don’t think it actually matters for this and it might get turned back on by windows update, but there are reports of it interfering with dual boot. On my machine I have to press F2 during startup during a very limited period to get to the bios setup. While there I can extend that time so it isn’t so hard to catch it. Now I don’t remember where exactly I found that on the NUC bios, but it waits 5 or 10 seconds for keyboard interrupt now on a power cycle. Turn off Fast Boot and make sure Secure boot is off. Make sure boot from USB is turned on. Here are some pictures of my bios screen. Not sure how useful they will be, but can’t hurt:

Download the Oracle Linux Distribution

I learned the hard way that the latest and greatest distribution (OL 8) is not supported for the Oracle database 19c!!!!!!! Go to the Oracle database documentation to find a table with supported Linux releases. At this time, it is OEL 7.4 with Unbreakable Kernel 5 (which is included by default). But I found out that when you auto update 7.4, you wind up with 7.8. I hope it will work because 7.4 unbreakable linux kernel did not support my network card. In any case I’m going to try 7.8

Go to the Oracle.com website and fumble around until you get to the dropdown menu that has On Premise Products/Software. Pick Oracle Linux. From there you can get to Download Oracle Linux which takes you to edelivery.oracle.com. From there you have to create an account or sign in. You do not need a support contract for this.

Search for Oracle Linux. Choose DLP: Oracle Linux 7.8. I know you want that tasty looking 8. Don’t do it. 7.8 is last supported release for the database. Add it to your cart and then go to Checkout on top right.

Pick x86 64bit for Platforms/Languages. Don’t freak out at the 14.0 GB, we will trim it down on the next step. Hit continue. Agree to the license. Get rid of the feedback popup. Uncheck everything by unchecking top level. Then go back and pick:

V995537-01.iso Oracle Linux Release 7 Update 8 for x86 (64 bit), 4.5 GB

That’s all you will need. Note that if you use the wget option, that shell script invokes wget with ask-password which the manual says will prompt for it, but I think it directed the prompt to stderr which was writing to a log file. I entered the password even though not prompted and it started downloading.

Download Oracle Grid and Oracle database 19c

Also get grid install and Oracle 19c from edelivery.oracle.com. You should be able to find them by searching. I did not take a screenshot of this. I have the files locally:

\-rwxrwxrwx. 1 nobody nobody 3059705302 Apr 29 10:55 LINUX.X64\_193000\_db\_home.zip 
\-rwxrwxrwx. 1 nobody nobody 2889184573 Apr 29 10:55 LINUX.X64\_193000\_grid\_home.zip

Burn ISO image to USB stick

Download rufus executable for windows. You don’t even have to install it. Just download it to the same place as your iso file. You can use that to create the bootable USB.

You can find plenty of instructions on the web about how to do that. The only non-obvious thing is choose FreeDOS for the format. I don’t know why, but it works. Install the iso file onto the USB.

Boot from USB stick and Install

If you set up the bios correctly, then it should see the USB stick first on boot up.

I picked Server with GUI as I want gnome desktop. I also added some optional products as shown on the right.

ENO1 network port

Allocating Disk Partitions

As mentioned above I had already created the partitions I wanted. You can do it here too. There are plenty of resources you can find about using the installer to do it. Here I’m just assigning the mount points to each partition and to the LVM.

Note that the /boot/efi partition is standard type EFI format, and /boot is standard type XFS format.

Swap is, well, swap format but is on the LVM partition. I made it twice the size of my 8GB of memory. Seems to be debatable how big it should be but 16GB is trivial.

The rest go on the LVM partition along with swap and are XFS. I probably overallocated / at 30GB, but I’m putting the /u01 directory there for the Oracle binaries and anything else I install. I’m slightly paranoid about running out of root space from past experience. Your mileage may vary. /home is 500GB because I have it available and sometimes you just need somewhere to put stuff.

Finish the Install

I gave myself an administrator account I can “sudo” from.

Then let the install rip.

When it finishes and wants to reboot, select that and give it a moment to shut down before pulling the USB stick. If you do it too soon, the shutdown hangs. If you it too late, it comes up on the installer boot image again. Either way just turn the machine off at that point. Pull the USB and it should boot to “grub” which will have the Unbreakable Linux kernel first in the boot list, with the RHL kernel next, a recovery image, and then the Windows image. You can rearrange the order later with “grubby” or directly in the grub config file. There are plenty of resources about that.

Remember the way I’ve done this, if I pull the plug out on the Lacie driver, the original boot loader on the internal drive should be picked up. This is as safe as it can be as long as I don’t ever do anything to that original internal Windows drive.

I booted from the unbreakable linux kernel and came up in gnome desktop, logged in as myself, then went to the Applications menu on the top left to get a gnonme-terminal and did:

sudo su -

Now I’m ready to rock.

Software Updates

From gnome desktop ran Applications/System Tools/Software Updates. Eventually it came up with 59 updates. Installed them. Rebooted.

xterm

sudo su - 
yum -y install xterm # Now I can start X on another machine, ssh into this one and launch an xterm 
# could use gnome-terminal but I'm old school.

Add EPEL repository

Put the following lines in /etc/yum.repos.d/epel-yum-ol7.repo

\[ol7\_epel\] name=Oracle Linux $releasever EPEL ($basearch) 
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/developer\_EPEL/$basearch/ 
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle 
gpgcheck=1 
enabled=1

\[root@linux2 ~\]# yum repolist 
Loaded plugins: langpacks, ulninfo 
ol7\_epel | 2.7 kB 00:00:00 
(1/3): ol7\_epel/x86\_64/updateinfo | 6.1 kB 00:00:00 
(2/3): ol7\_epel/x86\_64/group | 365 kB 00:00:02 
(3/3): ol7\_epel/x86\_64/primary\_db | 17 MB 00:01:01 
repo id repo name status 
ol7\_UEKR4/x86\_64 Latest Unbreakable Enterprise Kernel Release 4 for Oracle Linux 7Server (x86\_64) 96 
ol7\_epel/x86\_64 Oracle Linux 7Server EPEL (x86\_64) 31,547 
ol7\_latest/x86\_64 Oracle Linux 7Server Latest (x86\_64) 18,516 
repolist: 50,159

ntfs-3g

\[root@linux2 ~\]# yum -y install ntfs-3g 
...
Complete!

At this point the File Manager/Media Manager could mount the NTFS filesystem from the 3TB USB disk drive named “My Drive” on /media/run/*my login id*/something… A “df” command showed me the device was /dev/sdc1. There are other ways to find the device. In any case I dismounted that automatic drive using file manager and put the following entry into /etc/fstab:

/dev/sdc1   /mnt/g  ntfs-3g context="system\_u:object\_r:samba\_share\_t:s0",defaults,uid=nobody,gid=nobody 0   0

The context stuff may be required for samba, but that could have been a side effect of my experiment with security profiles. In any case it didn’t hurt anything, and with Secure Linux environment stuff you have to have that attribute for samba to share it. Also notice I’m mapping the username/groupname of all files on this drive as “nobody”. The default is “root” which is messy. Remember I am going to make this a public share.

Now create the mount point and mount it:

\[root@linux2 ~\]# mkdir /mnt/g 
\[root@linux2 ~\]# mount /mnt/g 
\[root@linux2 ~\]# ls -ld /mnt/g 
drwxrwxrwx. 1 nobody nobody 20480 May 5 11:47 /mnt/g 
\[root@linux2 ~\]#

samba

This reference was helpful: https://oracle-base.com/articles/linux/linux-samba-configuration

\[root@linux2 lee\]# yum -y install samba 
Loaded plugins: langpacks, ulninfo 
Package samba-4.10.4-10.el7.x86\_64 already installed and latest version 
Nothing to do 
\[root@linux2 lee\]# service smb start 
Redirecting to /bin/systemctl start smb.service 
\[root@linux2 lee\]# chkconfig smb on 
Note: Forwarding request to 'systemctl enable smb.service'. 
Created symlink from /etc/systemd/system/multi-user.target.wants/smb.service to /usr/lib/systemd/system/smb.service. 
\[root@linux2 lee\]#

Next/ I put the following in /etc/samba/smb.conf:

\[global\] 
workgroup = WORKGROUP 
netbios name = linux2 
security = user 
server string = Samba Server %v 
map to guest = bad user

\[g\] 
path = /mnt/g 
writable = yes 
browsable = yes 
guest ok = yes 
# guest account = lee 
# guest only = yes 
public = yes 
read only = no 
create mask = 0775 
directory mask = 0664

\[root@linux2 lee\]# service smb restart 
Redirecting to /bin/systemctl restart smb.service

But it still doesn’t work. The reason is the Linux firewall.

Firewall

I’m going to add the samba ports, but while I’m here I’ll also add Remote Dekstop Protocol ports for the step that follows samba. You’ll need to set your DISPLAY to point to your X server. If you are on gnome desktop, set it to :0.0. But in a window with your login shell or before doing switch user to root, remember to do xhost + so that the root shell can open the display.

\[root@linux2 lee\]# export DISPLAY=192.168.2.162:0.0 
\[root@linux2 lee\]# firewall-config

This brings up a gui for configuring the firewall on Linux. I selected eno1, which is my hardwired network interface, and hit the “Change Zone” button. I changed it from “public” to “home”. Next I added the samba services to home as shown:

Notice that was Runtime Configuration. Under Options menu choose “Runtime to Permanent” to save those changes. I also want to add “rdp” protocol so I can add xrdp and get the gnome desktop to my windows machine using Remote Desktop. To do that we have to add the ports which requires editing the Permanent config.

Change the Configuration: dropdown from Run time to Permanent and go to the Services tab. Hit the “+” at bottom:

Add “rdp” and click OK. Then select rdp from the list and click the Add button under Ports:

Also add same port with “udp”.

Select Zones tab. Select “home”. Add the new “rdp” service to home. Note that this is in the Permanent configuration. We will have to restart the service to pick it up. Under options select Reload Firewalld. After that you should be able to select rdp for the home zone. Now do options/Reload Firewalld again and you should see rdp checked for home zone.

It shouldn’t be needed, but you can restart the daemon to be sure it will come up next time:

\[root@linux2 lee\]# service firewalld stop 
Redirecting to /bin/systemctl stop firewalld.service 
\[root@linux2 lee\]# service firewalld start 
Redirecting to /bin/systemctl start firewalld.service

I fired firewall-config back up and now I see that under the “home” profile the new “rdp” service is allowed.

At this point samba is running and exposing my “/mnt/g” NTFS drive as the “g” share. The files on linux are owned by user “nobody”. It’s all read/write and my wife doesn’t yell at me any more about her pictures. It doesn’t matter if I’m booting windows or linux. In both cases using \\ip-address\g mounts the same drive!!!!!!!!!!!!!!!!

It is not secure you say. Hey. Ask me if I care.

Xrdp

I want to use Windows Remote Desktop client to get the gnome desktop on my PC so I’m not sitting in front of the TV to use it. Now that we have the EPEL repository configured, this is not that hard. Well, it was hard until I figured out the firewall issue that we addressed in the prior step, but now it is not hard.

\[root@linux2 lee\]# yum -y install xrdp tigervnc-server 
Loaded plugins: langpacks, ulninfo 
Resolving Dependencies 
--> Running transaction check 
---> Package tigervnc-server.x86\_64 0:1.8.0-19.0.1.el7 will be installed 
---> Package xrdp.x86\_64 1:0.9.5-1.el7 will be installed 
--> Processing Dependency: xorgxrdp for package: 1:xrdp-0.9.5-1.el7.x86\_64 
--> Running transaction check 
---> Package xorgxrdp.x86\_64 0:0.2.11-1.0.1.el7 will be installed 
--> Finished Dependency Resolution Dependencies Resolved 
======================================================================================================================================== 
Package Arch Version Repository Size 
======================================================================================================================================== 
Installing: 
tigervnc-server x86\_64 1.8.0-19.0.1.el7 ol7\_latest 216 k 
xrdp x86\_64 1:0.9.5-1.el7 ol7\_epel 412 k 
Installing for dependencies: 
xorgxrdp x86\_64 0.2.11-1.0.1.el7 ol7\_epel 63 k

Transaction Summary 
======================================================================================================================================== 
Install 2 Packages (+1 Dependent package)

Total download size: 691 k 
Installed size: 2.7 M 
Downloading packages: 
(1/3): xorgxrdp-0.2.11-1.0.1.el7.x86\_64.rpm | 63 kB 00:00:02 
(2/3): tigervnc-server-1.8.0-19.0.1.el7.x86\_64.rpm | 216 kB 00:00:03 
(3/3): xrdp-0.9.5-1.el7.x86\_64.rpm | 412 kB 00:00:03 
---------------------------------------------------------------------------------------------------------------------------------------- 
Total 178 kB/s | 691 kB 00:00:03 
Running transaction check 
Running transaction test 
Transaction test succeeded 
Running transaction 
Installing : xorgxrdp-0.2.11-1.0.1.el7.x86\_64 1/3 
Installing : 1:xrdp-0.9.5-1.el7.x86\_64 2/3 
Installing : tigervnc-server-1.8.0-19.0.1.el7.x86\_64 3/3 
Verifying : tigervnc-server-1.8.0-19.0.1.el7.x86\_64 1/3 
Verifying : xorgxrdp-0.2.11-1.0.1.el7.x86\_64 2/3 
Verifying : 1:xrdp-0.9.5-1.el7.x86\_64 3/3

Installed: 
tigervnc-server.x86\_64 0:1.8.0-19.0.1.el7 xrdp.x86\_64 1:0.9.5-1.el7

Dependency Installed: 
xorgxrdp.x86\_64 0:0.2.11-1.0.1.el7

Complete! 
\[root@linux2 lee\]# systemctl start xrdp 
\[root@linux2 lee\]# netstat -antup |grep xrdp 
tcp 0 0 0.0.0.0:3389 0.0.0.0:\* LISTEN 9515/xrdp 
tcp 0 0 127.0.0.1:3350 0.0.0.0:\* LISTEN 9514/xrdp-sesman 
\[root@linux2 lee\]# systemctl enable xrdp 
Created symlink from /etc/systemd/system/multi-user.target.wants/xrdp.service to /usr/lib/systemd/system/xrdp.service. 
\[root@linux2 lee\]#

Now from my laptop I run Windows Remote Desktop client entering the IP address of the Linux server (I’ll get around to naming service later) and it works!!!

We may get a secondary authentication request to enter the password for “keyring”. I am living with that for now.

We also get multiple authentication requests to create a color profile. Pffft. Google search turns up this:

https://unix.stackexchange.com/questions/417906/authentication-is-required-to-create-a-color-profile.

The answer that worked for me was to put the following in the file: /etc/polkit-1/localauthority/50-local.d/color.pkla

\[Allow colord for all users\] 
Identity=unix-user:\* 
Action=org.freedesktop.color-manager.create-device;org.freedesktop.color-manager.create-profile;org.freedesktop.color-manager.delete-device;org.freedesktop.color-manager.delete-profile;org.freedesktop.color-manager.modify-device;org.freedesktop.color-manager.modify-profile 
ResultAny=yes 
ResultInactive=yes 
ResultActive=yes

Now when I use Windows Remote Desktop I do not get a secondary prompt for a password (except sometimes for keyring). I have a remote desktop that looks just like the main gnome screen on the actual computer. I can still run an X windows server locally on my laptop and set the DISPLAY variable to have applications come that way, but most of the time just using the rdp connection desktop window works fine.

oracle-database-preinstall-19c-1.0-1.el7.x86_64.rpm

I have this file on the /mnt/g driver I was just mentioning. I suspect there is a way to get it with yum, but I already have it.

\# yum -y localinstall oracle-database-preinstall-19c-1.0-1.el7.x86\_64.rpm

It completed without issues. One of the things it did was create the “oracle” user account and most of the groups listed in the documentation; however, I’m going to install the grid software standalone per the 19c recommendation and so will be walking through that document adding the grid user and any missing groups.

\# yum update -y

Per this link https://oracle-base.com/articles/19c/oracle-db-19c-installation-on-oracle-linux-7, I ran through the following yum install commands. Some of them were not already present, but many were.

yum install -y bc 
yum install -y binutils 
yum install -y compat-libcap1 
yum install -y compat-libstdc++-33 
#yum install -y dtrace-modules 
#yum install -y dtrace-modules-headers 
#yum install -y dtrace-modules-provider-headers 
yum install -y dtrace-utils 
yum install -y elfutils-libelf 
yum install -y elfutils-libelf-devel 
yum install -y fontconfig-devel 
yum install -y glibc 
yum install -y glibc-devel 
yum install -y ksh 
yum install -y libaio 
yum install -y libaio-devel 
yum install -y libdtrace-ctf-devel 
yum install -y libXrender 
yum install -y libXrender-devel 
yum install -y libX11 
yum install -y libXau 
yum install -y libXi 
yum install -y libXtst 
yum install -y libgcc 
yum install -y librdmacm-devel 
yum install -y libstdc++ 
yum install -y libstdc++-devel 
yum install -y libxcb 
yum install -y make 
yum install -y net-tools # Clusterware 
yum install -y nfs-utils # ACFS 
yum install -y python # ACFS 
yum install -y python-configshell # ACFS 
yum install -y python-rtslib # ACFS 
yum install -y python-six # ACFS 
yum install -y targetcli # ACFS 
yum install -y smartmontools 
yum install -y sysstat 
yum install -y unixODBC

The Oracle Base document also recommends adding a password to the “oracle” account (and I presume “grid” account), setting secure Linux to permissive and stopping the firewall. Since these things are not discussed in the main documentation, I’m unsure about them. I’m going to skip and come back to them if I run into trouble that might be due to them.

Google Chrome

Although Firefox is fine, it has some quirks. Sometimes I would rather have Google Chrome. Installing it turned out to be ridiculously difficult, but I finally stumbled through some posts with enough information to figure it out.

\[root@linux2 Downloads\]# wget https://dl.google.com/linux/direct/google-chrome-stable\_current\_x86\_64.rpm 
\[root@linux2 Downloads\]# wget http://mirror.centos.org/centos/7/os/x86\_64/Packages/vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm 
\[root@linux2 Downloads\]# wget http://mirror.centos.org/centos/7/os/x86\_64/Packages/vulkan-1.1.97.0-1.el7.x86\_64.rpm 
\[root@linux2 Downloads\]# yum -y install libappindicator-gtk3-12.10.0-13.el7.x86\_64 
\[root@linux2 Downloads\]# yum -y install redhat-lsb libXScrnSaver 
\[root@linux2 Downloads\]# yum -y install liberation-fonts-1.07.2-16.el7.noarch 
\[root@linux2 Downloads\]# yum install -y liberation-narrow-fonts-1.07.2-16.el7.noarch 
\[root@linux2 Downloads\]# yum -y install vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm 
\[root@linux2 Downloads\]# yum install -y vulkan-1.1.97.0-1.el7.x86\_64.rpm 
\[root@linux2 Downloads\]# yum install -y google-chrome-stable\_current\_x86\_64.rpm

Installing Oracle Grid

Going through the checklist starting with section 2 (Checking and Configuring Server Hardware for Oracle Database). https://docs.oracle.com/en/database/oracle/oracle-database/19/ladbi/checking-server-hardware-and-memory-configuration.html#GUID-DC04ABB6-1822-444A-AB1B-8C306079439C

I have no issues.

We already did the preinstall rpm.

I do not have a support subscription, so skip Ksplice.

We need to install cvuqdisk, but it is in the grid package. We need to unzip that package. Before we can do that we need to create the “grid” user and some more groups that were not created automatically by the preinstall rpm. Then we can install the cvuqdisk rpm from there. We will come back to this.

The transparent huge page setting was taken care of for us I presume by the preinstall rpm:

\[root@linux2 yum.repos.d\]# cat /sys/kernel/mm/transparent\_hugepage/enabled 
always madvise \[never\]

My disk has the correct scheduler for ASM:

\[root@linux2 yum.repos.d\]# cat /sys/block/sda/queue/scheduler 
noop \[deadline\] cfq

Create users and groups for grid

\[root@linux2 ~\]# /usr/sbin/groupadd -g 54327 asmdba 
\[root@linux2 ~\]# /usr/sbin/groupadd -g 54328 asmoper 
\[root@linux2 ~\]# /usr/sbin/groupadd -g 54329 asmadmin

The rest of the groups already exist from the preinstall rpm.

\[root@linux2 ~\]# /usr/sbin/useradd -u 54331 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,asmoper,asmadmin grid

Although the “oracle” user already exists, edit /etc/group and add “oracle” to the list of users for the group “asmdba” or you can use “usermod” command as below.

\[root@linux2 ~\]# usermod -a -G asmdba oracle

For the users “grid” and “oracle” edit the .bash_profile and add the following two lines:

\[root@linux2 ~\]# vim ~grid/.bash\_profile ~oracle/.bash\_profile 2 files to edit

umask 022 
export ORACLE\_HOSTNAME=$(hostname)

Add the following lines to /etc/security/limits.conf:

oracle hard nofile 65536 
oracle soft stack 10240 
oracle hard stack 32768 
grid hard nofile 65536 
grid soft stack 10240 
grid hard stack 32768

Exit from the user grid/oracle and do “sudo su - grid” again to check the limits:

\[grid@linux2 ~\]$ ulimit -Sn 
1024 
\[grid@linux2 ~\]$ ulimit -Hn 
65536 
\[grid@linux2 ~\]$ ulimit -Su 
4096 
\[grid@linux2 ~\]$ ulimit -Hu 
61856 
\[grid@linux2 ~\]$ ulimit -Ss 
10240 
\[grid@linux2 ~\]$ ulimit -Hs 
32768

Copying grid install software

Create a directory for data files. Not sure we will use this and it is in the / partition, so we will see. Later I’m going to create a recovery area in /home.

\[root@linux2 ~\]# mkdir -p /u01/oradata 
\[root@linux2 ~\]# chown grid:oinstall /u01/oradata 
\[root@linux2 ~\]# chmod 775 /u01/oradata

I’m following section 10 “Installing Oracle Grid Infrastructure for a Standalone Server with a New Database Installation:

\[root@linux2 ~\]# mkdir -p /u01/app/oracle 
\[root@linux2 ~\]# mkdir -p /u01/app/oraInventory 
\[root@linux2 ~\]# chown -R oracle:oinstall /u01/app/oracle 
\[root@linux2 ~\]# chown -R grid:oinstall /u01/app/oraInventory 
\[root@linux2 ~\]# chmod -R 775 /u01/app

Note that the instructions say to do this as user “oracle”, but I’m going to be using user “grid”. I still want all of the directories above “grid” to be owned by “oracle”. I have my Oracle grid zip file on /mnt/g/lee. Yours will be somewhere else.

\[root@linux2 oracle\]# mkdir -p /u01/app/oracle/product/19.0.0/grid 
\[root@linux2 oracle\]# chmod -R 775 /u01/app/oracle/product 
\[root@linux2 oracle\]# chown -R oracle:oinstall /u01/app/oracle/product 
\[root@linux2 oracle\]# chown grid:oinstall /u01/app/oracle/product/19.0.0/grid 
\[root@linux2 bin\]# mkdir /u01/app/grid 
\[root@linux2 bin\]# chmod 775 /u01/app/grid 
\[root@linux2 bin\]# chown grid:oinstall /u01/app/grid

Now switch over to user grid.

\[grid@linux2 ~\]$ cd /u01/app/oracle/product/19.0.0/grid 
\[grid@linux2 grid\]$ unzip -q /mnt/g/lee/LINUX.X64\_193000\_grid\_home.zip

Now that we have the zip file unpacked we can go back to the step where we need the cvq rpm and install that as user “root”:

\[root@linux2 rpm\]# cd /u01/app/oracle/product/19.0.0/grid/cv/rpm 
\[root@linux2 rpm\]# CVUQDISK\_GRP=oinstall; export CVUQDISK\_GRP 
\[root@linux2 rpm\]# rpm -iv cvuqdisk-1.0.10-1.rpm 
Preparing packages... 
cvuqdisk-1.0.10-1.x86\_64

Configure asm disks with grid

This is done as “root”. Note that if you wind up needing to reinstall after this step (like if you run deinstall/deinstall), you may need to run a command to wipe out the disk header (dd if=/dev/zero of=/dev/sda4 bs=1024 count=100). Be very careful!!!!! But if you just follow the grid deinstall instructions without wiping out the header, and then try again, it won’t find the partitions.

Something I’m not sure about, but I had trouble otherwise, is to use “udev” to configure the ownership of the device files for my disk partitions. I see hints that suggest ASM should be taking care of this, but I just don’t know. I’m creating the following file:

\[root@linux2 ~\]# vim /etc/udev/rules.d/99-oracleasm.rules

KERNEL=="sda4", SUBSYSTEM=="block", OWNER="grid", GROUP="oinstall", MODE="0660" 
KERNEL=="sda5", SUBSYSTEM=="block", OWNER="grid", GROUP="oinstall", MODE="0660"

Then run the following command to verify the rule:

\# udevadm test $(udevadm info --query=path --name=sda4)

Then to reload the rules and trigger them:

\[root@linux2 ~\]# udevadm control --reload-rules 
\[root@linux2 ~\]# udevadm trigger 
\[root@linux2 ~\]# ls -lal /dev/sda\* 
brw-rw----. 1 root disk 8, 0 May 7 23:53 /dev/sda 
brw-rw----. 1 root disk 8, 1 May 7 23:53 /dev/sda1 
brw-rw----. 1 root disk 8, 2 May 7 23:53 /dev/sda2 
brw-rw----. 1 root disk 8, 3 May 7 23:53 /dev/sda3 
brw-rw----. 1 grid oinstall 8, 4 May 7 23:53 /dev/sda4 
brw-rw----. 1 grid oinstall 8, 5 May 7 23:53 /dev/sda5

That will hold up across reboots.

Now to label the partitions. We are back to using root.

\[root@linux2 rpm\]# export ORACLE\_HOME=/u01/app/oracle/product/19.0.0/grid 
\[root@linux2 rpm\]# export ORACLE\_BASE=/tmp 
\[root@linux2 rpm\]# cd /u01/app/oracle/product/19.0.0/grid/bin 
\[root@linux2 bin\]# fdisk -l # to find the partition name of my ASM allocation 
##### CAREFUL!!!! 
\[root@linux2 bin\]# ./asmcmd afd\_label DATA1 /dev/sda4 --init 
\[root@linux2 bin\]# ./asmcmd afd\_label DATA2 /dev/sda5 --init 
\[root@linux2 bin\]# ./asmcmd afd\_lslbl /dev/sda4 
-------------------------------------------------------------------------------- 
Label Duplicate Path 
================================================================================ 
DATA1 /dev/sda4 
\[root@linux2 bin\]# ./asmcmd afd\_lslbl /dev/sda5 
-------------------------------------------------------------------------------- 
Label Duplicate Path 
================================================================================ 
DATA2 /dev/sda5 
\[root@linux2 bin\]#

Now we switch to user “grid” (sudo su - grid). Recall that we already updated the profile setting environment as directed. If you get a message

_ERROR: Unable to verify the graphical display setup. This application requires X display. Make sure that xdpyinfo exist under PATH variable_

then you have two choices. You can run “xhost +” in a terminal of your original login user to allow X11 clients run by other users to connect to your gnome desktop (which I’m displaying via remote desktop protocol on my laptop), or you can set the DISPLAY variable to point to your local Xserver where you have done “xhost +” or started without authentication. Either way we need to set the DISPLAY variable. I echo the DISPLAY variable in a terminal window I launched from gnome and see “:11.0”. I’ll use that.

\[grid@linux2 grid\]$ pwd /u01/app/oracle/product/19.0.0/grid 
\[grid@linux2 grid\]$ export DISPLAY=:11.0 
\[grid@linux2 grid\]$ ./gridSetup.sh

Oops! The grid installer does not display correctly in my remote desktop client. Drats! It is probably an xresource or colormap thing or something. I don’t feel the urge to try to figure that out right now. OK, plan B. I run my local VcXsrv X11 server on my PC which I started with the “Disable access control” checked. Now I set my DISPLAY variable to point there and try again:

\[grid@linux2 grid\]$ export DISPLAY=192.168.2.162:0.0 
\[grid@linux2 grid\]$ ./gridSetup.sh

This takes a little while to prepare and give you the next screen. You can look in the /tmp directory and below for logs if you get worried. I found mine in /tmp/GridSetupActions2020-05-07_08-37-33AM/gridSetupActions2020-05-07_08-37-33AM.out.

I’m doing a standalone Server (Oracle Restart).

It eventually gives me this window showing the partitions I labeled with asmcmd. I’m chosing “External” redundancy because I have no need of mirroring on this setup (and no separate disk to put it on). I select my disks. I was going to try to use ASM Filter Driver, but with two partitions instead of one (or perhaps something else I did since I last tried it), it tells me it is not supported. Based on other posts I’ve seen, it looked like trouble anyway.

I don’t need separate passwords and am not worried about the password strength here so it scolds me.

I’m not registering with any cloud.

I’m happy with the user ids it finds.

The next screen shows the “grid_base” directory that if you keep it will cause trouble with oraInventory. Change this to the /u01/app/grid directory that we created in preparation.

Here is same screen with the directory name corrected.

If it suggests creating an oraInventory beneath the product directory, that will be bad. That is going to be under oracle base for the database installation. Make sure it is under /u01/app instead. If you had left the default grid base location earlier, you would. have had the wrong directory here which causes trouble on the Oracle install. It is fixable later though.

I’m going to run the configuration scripts manually rather than try to give it a way to do it with root.

Press the Install button.

\[root@linux2 bin\]# /u01/app/oraInventory/orainstRoot.sh 
Changing permissions of /u01/app/oraInventory. 
Adding read,write permissions for group. 
Removing read,write,execute permissions for world. 
# 
Changing groupname of /u01/app/oraInventory to oinstall. 
The execution of the script is complete.

And now the tricky one. Cross fingers. I got past the part I failed on OEL 8 and also on 7.8 when I had a greater than 2TB partition for ASM. This runs a while.

\[\[root@linux2 bin\]# /u01/app/oracle/product/19.0.0/grid/root.sh 
Performing root user operation. 
# 
The following environment variables are set as: 
ORACLE\_OWNER= grid 
ORACLE\_HOME= /u01/app/oracle/product/19.0.0/grid 
# 
Enter the full pathname of the local bin directory: \[/usr/local/bin\]: 
The contents of "dbhome" have not changed. No need to overwrite. 
The contents of "oraenv" have not changed. No need to overwrite. 
The contents of "coraenv" have not changed. No need to overwrite. 
# 
# 
Creating /etc/oratab file... 
Entries will be added to the /etc/oratab file as needed by 
Database Configuration Assistant when a database is created 
Finished running generic part of root script. 
Now product-specific root actions will be performed. 
Using configuration parameter file: /u01/app/oracle/product/19.0.0/grid/crs/install/crsconfig\_params 
The log of current session can be found at: /u01/app/grid/crsdata/linux2/crsconfig/roothas\_2020-05-08\_10-41-18AM.log 
LOCAL ADD MODE 
Creating OCR keys for user 'grid', privgrp 'oinstall'.. 
Operation successful. 
LOCAL ONLY MODE 
Successfully accumulated necessary OCR keys. 
Creating OCR keys for user 'root', privgrp 'root'.. 
Operation successful. 
CRS-4664: Node linux2 successfully pinned. 
2020/05/08 10:42:17 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service' 
# 
linux2 2020/05/08 10:45:13 /u01/app/grid/crsdata/linux2/olr/backup\_20200508\_104513.olr 724960844 
2020/05/08 10:45:15 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Hooray! Click OK back on the installer window.

This takes a long time. I see something going on in /tmp/hsperfdata_grid.

And now I get a failure.

Oracle Net Listener Startup: 
The information provided for this listener is currently in use by other software on this computer. 
Check the trace file for details: /u01/app/oracle/product/19.0.0/grid\_base/cfgtoollogs/netca/trace\_OraGI19Home1-2005079AM2649.log 
Oracle Net Services configuration failed. The exit code is 1 
Oracle Net Configuration Assistant failed.

And in the trace log file mentioned above I find:

\[main\] \[ 2020-05-07 10:01:43.447 EDT \] \[ConfigureListener.isPortFree:1292\] Creating ServerSocket on Port:1521, IP Address: linux2.localdomain/92.242.140.21 
\[main\] \[ 2020-05-07 10:01:43.447 EDT \] \[ConfigureListener.isPortFree:1328\] Cannot assign requested address (Bind failed) 
java.net.PlainSocketImpl.socketBind(Native Method) 
java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) 
java.net.ServerSocket.bind(ServerSocket.java:375) 
java.net.ServerSocket.<init>(ServerSocket.java:237) 
oracle.net.ca.ConfigureListener.isPortFree(ConfigureListener.java:1294) 
oracle.net.ca.ConfigureListener.startOrStopListener(ConfigureListener.java:1380) 
oracle.net.ca.ConfigureListener.typicalConfigure(ConfigureListener.java:369) 
oracle.net.ca.SilentConfigure.performSilentConfigure(SilentConfigure.java:212) 
oracle.net.ca.InitialSetup.<init>(NetCA.java:4325) 
oracle.net.ca.NetCA.main(NetCA.java:460) 
\[main\] \[ 2020-05-07 10:01:43.447 EDT \] \[ConfigureListener.isPortFree:1341\] Returning is Port 1521 free: false

After a little googling and remembering advice I had read in the Oracle Base document, I found I had failed to update /etc/hosts with the name of this computer and the IP address. I added it to /etc/hosts:

\[root@linux2 bin\]# cat /etc/hosts 
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 
192.168.2.81 linux2.localdomain linux2

After this it completed the net configuration assistant, but failed the ASM assistant:

\[FATAL\] \[DBT-30002\] Disk group DATA creation failed. 
ORA-15018: diskgroup cannot be created 
ORA-15031: disk specification '/dev/sda5' matches no disks 
ORA-15025: could not open disk "/dev/sda5" 
ORA-27041: unable to open file 
Automatic Storage Management Configuration Assistant failed.

Some google searching leads me to believe that we have to manually assign ownership of the device files for the ASM partitions. Wish that had been mentioned in the documentation (or if it was, that I had noticed it). I proceeded with the “udev” rule to change the ownership of the device files as mentioned above. You should have already done that. At that point the /dev/sda[45] files were changed to grid:oinstall which takes care of it.

Pressed continue and it finished. Yeaaaaaa!

Testing Grid Installation

The next section in the install document is Testing the Oracle Automatic Storage Management Installation. I presume the sid I want is +ASM. When I look in the grid home dbs directory I see dbs/init.ora with a db_name=’ORCL’, but in the dbs directory I see some files with ‘+ASM’ in the name. I have zero experience with grid yet, so I’m guessing.

\[grid@linux2 grid\]$ export ORACLE\_HOME=/u01/app/oracle/product/19.0.0/grid 
\[grid@linux2 grid\]$ export ORACLE\_SID=+ASM 
\[grid@linux2 grid\]$ $ORACLE\_HOME/bin/asmcmd lsdg 
State Type Rebal Sector Logical\_Sector Block AU Total\_MB Free\_MB Req\_mir\_free\_MB Usable\_file\_MB Offline\_d 
MOUNTED EXTERN N 512 512 4096 4194304 2097152 2097040 0 2097040

Must have guessed right. That looks like both my 1 TB partitions were combined into a single group as expected. The ASM instance must be runing (a ps listing shows processes with +ASM in the name as well as a tnslsnr).

I can also fire up $ORACLE_HOME_/bin/asmca_ and look around.

The next section of the document informs us that every time we do any OS upgrades we need to relink the grid executables and provides some instructions. That’s beyond the scope for now, but try to remember this for the future.

I won’t be using ACFS which is storage on stuff other than the database. And I don’t have access to support, so I don’t have any patches to apply.

That’s it for grid.

Installing Oracle Database

I already have to directories listed in step 2 of running the wizard, so all I need to do is create dbhome_1. I sudo su - oracle, and unpack my zip file.

\[oracle@linux2 19.0.0\]$ mkdir -p /u01/app/oracle/product/19.0.0/dbhome\_1 
\[oracle@linux2 19.0.0\]$ cd /u01/app/oracle/product/19.0.0/dbhome\_1 
\[oracle@linux2 dbhome\_1\]$ unzip -q /mnt/g/lee/LINUX.X64\_193000\_db\_home.zip 
\[oracle@linux2 dbhome\_1\]$ export DISPLAY=192.168.2.162:0.0 # you may be going local, but you will still need to set it and perhaps do xhost+ first
\[oracle@linux2 dbhome\_1\]$ ./runInstaller 
Launching Oracle Database Setup Wizard...

I’ll be creating the single instance database.

After looking around a little bit I decided the “Desktop class” installation would be too limiting. I want to be able to play here which includes connecting from other machines. I picked “Server class”.

Again, because I’m using this database to learn about big iron stuff, I picked Enterprise Edition. I don’t “need” it, but I want to be able to explore it. YMMV.

Note that the next section describes an issue I subsequently fixed in the grid install process. I had made a mistake on the oracle inventory location. I leave this here in case you find yourself in the same situation.

Hmm. Is this a permission issue?

\[root@linux2 19.0.0\]# ls -l 
total 8 
drwxr-xr-x. 68 oracle oinstall 4096 May 7 19:27 dbhome\_1 
drwxr-x---. 85 root oinstall 4096 May 7 09:18 grid 
drwxr-xr-x. 7 grid oinstall 84 May 7 10:32 grid\_base 
drwxrwx---. 5 grid oinstall 92 May 7 19:20 oraInventory 
\[root@linux2 19.0.0\]# chown grid grid 
\[root@linux2 19.0.0\]# chmod 775 \* 
\[root@linux2 19.0.0\]# ls -l 
total 8 
drwxrwxr-x. 68 oracle oinstall 4096 May 7 19:27 dbhome\_1 
drwxrwxr-x. 85 grid oinstall 4096 May 7 09:18 grid 
drwxrwxr-x. 7 grid oinstall 84 May 7 10:32 grid\_base 
drwxrwxr-x. 5 grid oinstall 92 May 7 19:20 oraInventory

I had to exit the installer and start it again. This time when I got to the step for Oracle base it did not complain immediately, but did give a popup when I continued.

I see plenty of questions about this but don’t see an explanation I can follow other than the grid install was supposed to put the inventory in /u01/app/oraInventory. I’m not sure why it did not the first time (In my latest attempt as documented in this blog I correctly located it at /u01/app/oraInventory). I found this link on how to move oraInventory (https://logic.edchen.org/how-to-move-central-inventory/). I changed /etc/OraInst.loc:

\[root@linux2 19.0.0\]# cat /etc/oraInst.loc 
#inventory\_loc=/u01/app/oracle/product/19.0.0/oraInventory 
inventory\_loc=/u01/app/oraInventory 
inst\_group=oinstall

The existing files are owned by grid, but since the group is oinstall we should be good. You may need to make sure any existing log directories are group writeable first. As user grid::

\[grid@linux2 oraInventory\]$ cp -rp /u01/app/oracle/product/19.0.0/oraInventory /u01/app 
\[grid@linux2 oraInventory\]$ pwd 
/u01/app/oraInventory 
\[grid@linux2 oraInventory\]$ cp /etc/oraInst.loc . 
\[grid@linux2 grid\]$ cd/u01/app/oracle/product/19.0.0/grid
\[grid@linux2 grid\]$ cp /etc/oraInst.loc .

Now back as oracle user I try the installer again. We are past that hurdle and continuing. Note that since doing this I did deinstall of both the database and grid and started over.

I’m customizing my database name and also going to create it as a container database. Total overkill, but I want to play with pluggable database containers.

It defaults to NOT enabling automatic memory management because there is more than 4GB of memory. Not sure I understand that. Note that you have to move the slider to get close to 4.096 before you can edit the text box. I don’t know why.

Although I have no plans to use anything but ascii, I’ll go with the default because that is what most large corporations do, I think.

I’ll go ahead and install the sample schemas. They often make a good base for doing examples.

I will be using ASM.

No cloud for me.

I will not be putting anything on this database I can’t afford to lose. In fact, I expect to tear it down and start over many times. But I tried it without recovery being set and wound up with an error later. I’ll enable recovery.

It found my ASM disc group.

Yes, I’m a bad person with passwords on this database.

Just like with the grid install, I prefer to run the root scripts myself separately.

\[root@linux2 ~\]# /u01/app/oracle/product/19.0.0/dbhome\_1/root.sh 
Performing root user operation. 
# 
The following environment variables are set as: 
ORACLE\_OWNER= oracle 
ORACLE\_HOME= /u01/app/oracle/product/19.0.0/dbhome\_1 
# 
Enter the full pathname of the local bin directory: \[/usr/local/bin\]: 
The contents of "dbhome" have not changed. No need to overwrite. 
The contents of "oraenv" have not changed. No need to overwrite. 
The contents of "coraenv" have not changed. No need to overwrite. 
# 
Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created
Finished running generic part of root script. 
Now product-specific root actions will be performed. 
Oracle Trace File Analyzer (TFA - Standalone Mode) is available at : /u01/app/oracle/product/19.0.0/dbhome\_1/bin/tfactl 
# 
Note : 
1. tfactl will use TFA Service if that service is running and user has been granted access 
2. tfactl will configure TFA Standalone Mode only if user has no access to TFA Service or TFA is not installed 
# 
\[root@linux2 ~\]#

Pressed the OK button on the root dialog.

Running the command “iotop -o -d 5” I can see the database is writing to disk. At least that is better than just watching the installer sit there with “In Progress”. As a side note the Applications/System Tools/System Monitor app on gnome is not very impressive. There has to be a better tool.

I found some logs I could watch

[ oracle@linux2 cfgtoollogs]$ pwd /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs [oracle@linux2 cfgtoollogs]$ tail -f oracle.server_2020-05-08_09-30-54-AM.log INFO: Skipping line: 38% complete INFO: Skipping line: 38% complete INFO: Skipping line: 42% complete INFO: Skipping line: 42% complete INFO: Skipping line: 45% complete INFO: Skipping line: 45% complete INFO: Skipping line: 48% complete INFO: Skipping line: 48% complete INFO: Skipping line: Completing Database Creation INFO: Skipping line: Completing Database Creation

Finally it finishes.

But Firefox complains with that address - something about security. I found links to solutions having to do with XDB or private certificates or some such. I spent several hours looking without success. Someone with access to support might be able to figure it out. I might come back to that. I don’t need it though.

The database is installed and functioning. Shutting down and starting up Linux brings the database down and up gracefully.

I configured the .bash_profile of the oracle user with the following lines.

export PATH umask 022 export ORACLE_HOSTNAME=$(hostname) export ORAENV_ASK=NO export ORACLE_SID=leedb . oraenv -s

Now I can do:

sqlplus / as sysdba 
# or 
sqlplus system

without having to set any more variables.

I configured the .bash_profile of the grid user with the following lines:

umask 022 
export ORACLE\_HOSTNAME=$(hostname) 
export ORACLE\_HOME=/u01/app/oracle/product/19.0.0/grid 
export ORACLE\_SID="+ASM" 
PATH=${PATH}:${ORACLE\_HOME}/bin

Now when I log in as user grid I can issue srvctl commands. See below.

Starting and Stopping

If you have not already done so, read https://docs.oracle.com/en/database/oracle/oracle-database/19/admin/configuring-automatic-restart-of-an-oracle-database.html#GUID-EA0104A2-B256-4866-9C8A-9B69F8D0F24E

If you use sqlplus to shutdown your database you will be in for a surprise when you try to start it again. You can recover. After thinking I had lost the diskgroup I found I could get it all going again by logging in as “grid” and running a few commands:

srvctl start diskgroup -diskgroup data 
srvctl start database -db leedb

In general you can let the normal Linux shudtown and startup take care of things. When you are ready, you need to learn about srvctl which is kind of the point of setting up the gird “restart” structure.

Sqldeveloper

You may be able to use yum to get sqldeveloper, but I downloaded the rpm. Before installing it you need the java sdk.

\[root@linux2 jvm\]# yum install java-1.8.0-openjdk-devel ... 
\[root@linux2 jvm\]# yum install sqldeveloper-19.1.0.094.2042.noarch.rpm

It installs in /usr/local/bin. The first time you run sqldeveloper it prompts for the java sdk location. There are some soft links that take care of adjusting for upgrades. Answer with:

/usr/lib/jvm/java-openjdk

which will point to the latest and greatest that you just installed.

Conclusion

This was much, much harder than it should have been. I spent more hours on this than I am willing to admit. Granted, I was learning some Linux things I didn’t know and some Oracle administration things I didn’t know as I was going, but even with that background this was a very hard slog.

Access to Oracle support would have made it a little easier, but not that much. I do not know why it is such a cluster (pun intended).

I hope a few people might find this useful.