Friday, May 16, 2025

Oracle Documentation and Advantages of Reading Other Resources

After I published the first post of the series (#1, #2) I received a private request to talk a bit more about 11.2-to-19c upgrade. Considering that frankly speaking that's the most common upgrade case nowadays (well, one long-term release to the other one!) I can definitely understand the importance of real-world experience for anybody who plans to do the same.

So, here is the story - or to be precise two stories.
  1. Production/Stage/External Test - these ones were handled by our partner DBAs, but the idea was pretty straightforward: copy cold backup to the new environment + run in-place upgrade (non-CDB). Same OS (Linux-to-Linux), preserving of character sets etc. I.e. everything is well-documented and clear. So, nothing much to write about.
  2. Development and Internal Test - that one was a real head-scratcher that I'd like to share with everybody. The task was as follows:
    • Starting points: Windows 2008 Server with 11.2.0.4 database and Win1252 charset, Windows 2012 Server with 19c database win Win1252 charset (non-CDB). Each DB is about 200 GB
    • Required destination: Amazon AWS Linux instance with 19c database with two environments as PDBs that would allow creating ALUTF32 PDBs if needed.
As you can see, a lot of transformation and less-then-common transfer path. One of the reasons - you just can't install 19c database to run in-place upgrade using AutoUpgrade and other nice things on Windows 2008 server. So, with the help of our system administrators I came up with the plan that eventually lead to success. And that's where "other-then-documentation" resources really helped - some pieces of knowledge I took from Tim Hall's Oracle-Base, some pieces are from Mike Dietrich, some pieces from other people just posting their ideas and experiences in their blogs and various online resources. Overall, without the community I would experiment forever! And that's why I like so much to be part of this knowledge-sharing environment - and try to contribute myself.

I started from the bottom -  discussed with sysadmins a good and reliable way to moving something to AWS. The answer I've got was interesting: there was a simple  process to port existing virtual machine from our internal cloud to AWS. That made my life much simple - it meant that we could build a local VM, play with it as much as needed (with a very easy way to create multiple saverpoints/restore to any point etc.) and eventually declare it ready to be transferred. Ok, admins got Oracle Linux ISO and made me in vSphere a new server - now I've got a destination.

Next step was to see whether my understanding of OS internal was true and it was possible to move PDB from Windows to Linux without any major hustle... To do that I first converted my 19c instance from non-CDB to a PDB within newly created PDB. That was just a couple of commands:

-- non-CDB instance
shutdown;
startup open read only
/
BEGIN
  DBMS_PDB.DESCRIBE(
    pdb_descr_file => 'f:\oradata\test01.xml');
END;
/
shutdown immediate
/

-- CDB instance
CREATE PLUGGABLE DATABASE test01 
using 'f:\oradata\test01.xml'
NOCOPY 
tempfile reuse 
/
ALTER SESSION SET CONTAINER=test01;
@$ORACLE_HOME/rdbms/admin/noncdb_to_pdb.sql;
ALTER PLUGGABLE DATABASE OPEN;

Now it was time to move PDB to Linux box, that already had CDB configured as AL32UTF... And after not finding the answer on whether file_name_convert operator can change Windows-style file name to Linux-style I just went into moveText.xml file and manually updated all locations to which I transferred  PDB files (btw, later I tested the operator - and I couldn't make it do what I wanted, so, hint: sometimes manual approach is your choice ;) )

-- Windows CDB
ALTER PLUGGABLE DATABASE test01 CLOSE;
ALTER PLUGGABLE DATABASE test01 UNPLUG INTO 'F:\moveTest.xml';

-- Linux CDB
CREATE PLUGGABLE DATABASE test01 
using '/home/oracle/test01_updated.xml' -- Linux locations
NOCOPY 
tempfile reuse 
/
ALTER SESSION SET CONTAINER=test01;
ALTER PLUGGABLE DATABASE OPEN;

The last command finished - but with the warning. It was time to review existing violations

select *
from pdb_plug_in_violations
where name = 'TEST01'
and status !='RESOLVED'

The most only errors I found was the mismatch of patches - and that was simply fixed by running datapatch - verbose on the Linux side: the system easily rolled back all of the incorrect patches within PDB and placed what was needed. Seemingly the system was ready to operate - I could query data, run some sample procedures... But the application didn't start! After some digging I found a somewhat unexpected reason - even though you could move Windows PDB to Linux, if you are using PL/SQL objects with Native compilation - you have to recompile everything (well, makes sense, different platforms = different low-level code!). The following loop did that:

 begin
    for c in (
        select owner, name, type
        from dba_plsql_object_settings
        where plsql_code_type !='INTERPRETED'
        )
    loop
        begin
            if c.type = 'PACKAGE BODY' then
                execute immediate 'alter '||c.type||' '                                   ||c.owner||'.'||c.name||' compile body';
            else
                execute immediate 'alter '||c.type||' '                                   ||c.owner||'.'||c.name||' compile';
            end if;
        exception
            when others then null;
        end;
    end loop;
end;

That was easy enough - so I decided to replicate exactly the same pattern for the main development database (that was 11.2.0.4). So, in my Test database I dropped PDB Test01 with all datafiles (I had to do it because Dev files names overlapped with Test01 and I didn't want to shuffle anything around) and used good-old-and tried approach:
  1. Original DB
    • Create PFILE from SPFILE
    • Run preupgrade on the source and follow its recommendations
    • Shutdown the instance
    • Transfer files to the new location preserving the file structure (as for the cold backup):
      • ORADATA\ora11g folder
      • FAST_RECOVERY AREA\ora11g folder with all corresponding directories
      • PFILE
      • Control files from other locations (btw, you can integrate them into the same folder by modifying PFILE)
  2. Server with 19c
    • Create new instance:
      • orapwd file=? password=? format=12 (lower format if you want simple password for now)
      • oradim
      • startup upgrade
    • Upgrade via dbupgrade utility
    • Run post-fixes, including DST upgrade
    • Recompile whatever needed and resolve obscure cases if exist (Java classes, SDO_Geometry etc.)
Now I was pretty exactly where I started with Test environment - non-CDB instance + empty container. So, the following steps are exactly above: convert instance into PDB, plug it in as Dev01, upgrade, fix PL/SQL native-compiled objects. That's all, folks - two environments became one on the different platform! After that it was just lift-and-shift to AWS - and we've got our new environment.

Summary: if you can use somebody else's experience - use it! Oracle documentation, especially about upgrades is huge, but sometimes you may be looking for something very specific and it may be hard to detect exactly your use-case in the superset of the knowledge. 

P.s. VM Snapshots are your saviors (that's why I love the cloud). You can't imagine how many various issues I've got in process - and without snapshot recovery it would've take wa-a-a-a-y longer to do anything meaningful.

Wednesday, May 14, 2025

Oracle Documentation and Dangers of Not Reading It

In addition to yesterday's post - a bit of old-timer knowledge share: if you see some features for years, you have a tendency to assume how they work. And often enough to your own surprise "the owls are not what they seem" (c) :)

My favorite story here is about hierarchical queries with CONNECT-BY clause. For years I was wondering why every generation of developers was having various performance issues writing such queries... Recently, while working on the presentation about recursive structures in the database I came to realization of the following: people look up code snippets without going any further!

Let's take a basic problem based on HR schema - list all employees working for the specified department in the hierarchical order starting from its manager. A lot of junior developer would write something like that:

select *
from hr.employees
where department_id = 100
start with job_id like '%MGR%' 
connect by prior employee_id = manager_id

Result of the query would be functionally correct - so, developers would start using that pattern... And eventually (when they would start working with large data sets!) that code becomes the source of the major performance degradation! Here is why - let's take a look at the explain plan:

------------------------------------------------------------------------------
| Id  | Operation                                | Name      | Rows  | Bytes |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                         |           |    34 |  4522 |
|*  1 |  FILTER                                  |           |       |       |
|*  2 |   CONNECT BY NO FILTERING WITH START-WITH|           |       |       |
|   3 |    TABLE ACCESS FULL                     | EMPLOYEES |   107 |  7383 |
------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
1 - filter("EMPLOYEES"."DEPARTMENT_ID"=100)
2 - access("EMPLOYEES"."MANAGER_ID"=PRIOR "EMPLOYEES"."EMPLOYEE_ID")
       filter("JOB_ID" LIKE '%MGR%')

Even though we have lots of indexes on the table, we've got FTS - and if we look closer, we will notice that our department filter was applied at the very end. It seems that Oracle first walked the whole table hierarchically - and after that threw out all rows that do not belong to the department #100!

And indeed that's exactly what Oracle documentation has been telling us for many years:


Indeed non-Join predicates are evaluated only afterwards. In a lot of cases it indeed makes sense - depending on the conditions. But in THIS case, where we know (and that's the key knowledge that Oracle is missing) that hierarchies never cross departments - although we forgot to tell Oracle about it! So, no wonder, the result is les then optimal.

How to fix it? There are two approaches to solve this problem that both significantly improve performance, but provide slightly different executions plans.

Option #1 is to use in-line views (to be fair, that's my favorite approach, because it makes code way more readable):

select *
from (select *
      from hr.employees
      where department_id = 100)
start with job_id like '%MGR%'
connect by prior employee_id = manager_id

-------------------------------------------------------------------------------------
| Id  | Operation                               | Name              | Rows  | Bytes |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                        |                   |     2 |   266 |
|*  1 |  CONNECT BY NO FILTERING WITH START-WITH|                   |       |       |
|   2 |   TABLE ACCESS BY INDEX ROWID BATCHED   | EMPLOYEES         |     6 |   414 |
|*  3 |    INDEX RANGE SCAN                     | EMP_DEPARTMENT_IX |     6 |       |
-------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   1 - access("EMPLOYEES"."MANAGER_ID"=PRIOR "EMPLOYEES"."EMPLOYEE_ID")
       filter("EMPLOYEES"."JOB_ID" LIKE '%MGR%')
   3 - access("DEPARTMENT_ID"=100)

As you can see here, explicitly filtering the department #100 in the in-line view immediately dropped number of processed rows and bytes - exactly as needed.

Option #2 is to tell Oracle to start only from rows that belong to the department #100:

select *
from hr.employees
start with job_id like '%MGR%' and department_id = 100
connect by prior employee_id = manager_id

-----------------------------------------------------------------------------------
| Id  | Operation                             | Name              | Rows  | Bytes |
-----------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                   |     7 |   931 |
|*  1 |  CONNECT BY WITH FILTERING            |                   |       |       |
|*  2 |   TABLE ACCESS BY INDEX ROWID BATCHED | EMPLOYEES         |     1 |    69 |
|*  3 |    INDEX RANGE SCAN                   | EMP_DEPARTMENT_IX |     6 |       |
|   4 |   NESTED LOOPS                        |                   |     6 |   492 |
|   5 |    CONNECT BY PUMP                    |                   |       |       |
|   6 |    TABLE ACCESS BY INDEX ROWID BATCHED| EMPLOYEES         |     6 |   414 |
|*  7 |     INDEX RANGE SCAN                  | EMP_MANAGER_IX    |     6 |       |
-----------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
--------------------------------------------------- 
   1 - access("EMPLOYEES"."MANAGER_ID"=PRIOR "EMPLOYEES"."EMPLOYEE_ID")
   2 - filter("JOB_ID" LIKE '%MGR%')
   3 - access("DEPARTMENT_ID"=100)
   7 - access("connect$_by$_pump$_002"."prior employee_id "="MANAGER_ID")

It has slightly higher cost an somewhat more complex execution plan - but that way you still have better performance than FTS alternative.

Summary: please, beware - solutions that work may not scale if you don't understand what is going under the hood! Read those manuals, please!









Tuesday, May 13, 2025

Oracle Documentation and Dangers of the AI

Earlier today I presented a webinar for NYOUG on one of my favorite topics - building SQL queries on the fly (btw, if anybody interested - slides and video will recording will be available a bit later):
https://nyoug.org/event/dynamic-mechanisms-of-query-building/

And during the presentation a recent in-house story came to my mind - based on my own remark. I said something like: "If you are using advanced functionality, you really need to know the features you are planning to use - not just copy/paste code snippets from ChatGPT or any other preferred AI tool" 

The statement above is perfectly true - and I even got an illustration (kind-of) for it based on the true "mini-scare" that happened in our office.

Here is the story. For many years a lot of my friends were laughing at me when I was telling that one of the biggest (government) projects that is being supported by Dulcian is still on Oracle 11.2.0.4. Finally this year our partners responsible for the administering the system got it to 19c. All is good - but for some internal reasons the production environment is still running with  "compatible=11.2.0.4.0"!

Of course, for us (i.e. development shop) it is a major restriction - and I've been very vocal about finally moving that parameter to "19.0.0". Couple of weeks ago in the internal communication I've got a bit panicky email from one of my colleagues: "Depending on how I ask an AI get different answers whether downgrade is possible! Help!" + a couple of screenshots:

#1 Gemini (Google)




#2 ChatGPT

As you can see, one platform says clear "NO" and the other one says that it is clearly possible, but there may be some impact that needs to be evaluated.

After hours of research I can say the following - Gemini is closer to the real answer than ChatGPT. Even though there are empirical evidences that it is possible in some circumstances (different combinations of versions) to lower compatibility parameter, it is strongly not recommended. So much not recommended that in 19c docs it is explicitly written that you cannot go back to 11.2.0.4 at all:



Summary: be VERY careful when you use various AI tools - it could be a good starting point, but without going to the core references you cannot be sure that you are getting real answers. Buyers, beware! :)